- All Implemented Interfaces:
Unwrappable<MergePolicy>
BPIndexReorderer
. When
reordering doesn't have enough RAM, it simply skips reordering in order not to fail the merge. So
make sure to give enough RAM to your BPIndexReorderer
via BPIndexReorderer.setRAMBudgetMB(double)
.-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.index.MergePolicy
MergePolicy.MergeAbortedException, MergePolicy.MergeContext, MergePolicy.MergeException, MergePolicy.MergeSpecification, MergePolicy.OneMerge, MergePolicy.OneMergeProgress
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate int
private float
(package private) static final String
Whether a segment has been reordered.private final BPIndexReorderer
Fields inherited from class org.apache.lucene.index.FilterMergePolicy
in
Fields inherited from class org.apache.lucene.index.MergePolicy
DEFAULT_MAX_CFS_SEGMENT_SIZE, DEFAULT_NO_CFS_RATIO, maxCFSSegmentSize, noCFSRatio
-
Constructor Summary
ConstructorsConstructorDescriptionBPReorderingMergePolicy
(MergePolicy in, BPIndexReorderer reorderer) Sole constructor. -
Method Summary
Modifier and TypeMethodDescriptionfindForcedDeletesMerges
(SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) Determine what set of merge operations is necessary in order to expunge all deletes from the index.findForcedMerges
(SegmentInfos segmentInfos, int maxSegmentCount, Map<SegmentCommitInfo, Boolean> segmentsToMerge, MergePolicy.MergeContext mergeContext) Determine what set of merge operations is necessary in order to merge to<=
the specified segment count.findFullFlushMerges
(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) Identifies merges that we want to execute (synchronously) on commit.findMerges
(CodecReader... readers) Define the set of merge operations to perform on provided codec readers inIndexWriter.addIndexes(CodecReader...)
.findMerges
(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) Determine what set of merge operations are now necessary on the index.private MergePolicy.MergeSpecification
maybeReorder
(MergePolicy.MergeSpecification spec, boolean forced, SegmentInfos infos) void
setMinNaturalMergeNumDocs
(int minNaturalMergeNumDocs) Set the minimum number of docs that a merge must have for the resulting segment to be reordered.void
setMinNaturalMergeRatioFromBiggestSegment
(float minNaturalMergeRatioFromBiggestSegment) Set the minimum number of docs that a merge must have for the resulting segment to be reordered, as a ratio of the total number of documents of the current biggest segment in the index.Methods inherited from class org.apache.lucene.index.FilterMergePolicy
getMaxCFSSegmentSizeMB, getNoCFSRatio, keepFullyDeletedSegment, maxFullFlushMergeSize, numDeletesToMerge, setMaxCFSSegmentSizeMB, setNoCFSRatio, size, toString, unwrap, useCompoundFile
Methods inherited from class org.apache.lucene.index.MergePolicy
assertDelCount, isMerged, message, segString, verbose
-
Field Details
-
REORDERED
Whether a segment has been reordered.- See Also:
-
reorderer
-
minNaturalMergeNumDocs
private int minNaturalMergeNumDocs -
minNaturalMergeRatioFromBiggestSegment
private float minNaturalMergeRatioFromBiggestSegment
-
-
Constructor Details
-
BPReorderingMergePolicy
Sole constructor. It takes the merge policy that should be used to compute merges, and will then reorder doc IDs from all merges above the configured minimum doc count, as well as all forced merges.If you wish to only run reordering upon forced merges, pass
Integer.MAX_VALUE
as aminNaturalMergeNumDocs
. Otherwise a default value of2^18 = 262,144
is suggested. This should help retain merging optimizations on small merges while reordering the larger segments that are important for good search performance.- Parameters:
in
- the merge policy to use to compute mergesreorderer
- theBPIndexReorderer
to use to renumber doc IDs
-
-
Method Details
-
setMinNaturalMergeNumDocs
public void setMinNaturalMergeNumDocs(int minNaturalMergeNumDocs) Set the minimum number of docs that a merge must have for the resulting segment to be reordered. -
setMinNaturalMergeRatioFromBiggestSegment
public void setMinNaturalMergeRatioFromBiggestSegment(float minNaturalMergeRatioFromBiggestSegment) Set the minimum number of docs that a merge must have for the resulting segment to be reordered, as a ratio of the total number of documents of the current biggest segment in the index. This parameter helps only enable reordering on segments that are large enough that they will significantly contribute to overall search performance. -
maybeReorder
private MergePolicy.MergeSpecification maybeReorder(MergePolicy.MergeSpecification spec, boolean forced, SegmentInfos infos) -
findMerges
public MergePolicy.MergeSpecification findMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) throws IOException Description copied from class:MergePolicy
Determine what set of merge operations are now necessary on the index.IndexWriter
calls this whenever there is a change to the segments. This call is always synchronized on theIndexWriter
instance so only one thread at a time will call this method.- Overrides:
findMerges
in classFilterMergePolicy
- Parameters:
mergeTrigger
- the event that triggered the mergesegmentInfos
- the total set of segments in the indexmergeContext
- the IndexWriter to find the merges on- Throws:
IOException
-
findForcedMerges
public MergePolicy.MergeSpecification findForcedMerges(SegmentInfos segmentInfos, int maxSegmentCount, Map<SegmentCommitInfo, Boolean> segmentsToMerge, MergePolicy.MergeContext mergeContext) throws IOExceptionDescription copied from class:MergePolicy
Determine what set of merge operations is necessary in order to merge to<=
the specified segment count.IndexWriter
calls this when itsIndexWriter.forceMerge(int)
method is called. This call is always synchronized on theIndexWriter
instance so only one thread at a time will call this method.- Overrides:
findForcedMerges
in classFilterMergePolicy
- Parameters:
segmentInfos
- the total set of segments in the indexmaxSegmentCount
- requested maximum number of segments in the indexsegmentsToMerge
- contains the specific SegmentInfo instances that must be merged away. This may be a subset of all SegmentInfos. If the value is True for a given SegmentInfo, that means this segment was an original segment present in the to-be-merged index; else, it was a segment produced by a cascaded merge.mergeContext
- the MergeContext to find the merges on- Throws:
IOException
-
findForcedDeletesMerges
public MergePolicy.MergeSpecification findForcedDeletesMerges(SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) throws IOException Description copied from class:MergePolicy
Determine what set of merge operations is necessary in order to expunge all deletes from the index.- Overrides:
findForcedDeletesMerges
in classFilterMergePolicy
- Parameters:
segmentInfos
- the total set of segments in the indexmergeContext
- the MergeContext to find the merges on- Throws:
IOException
-
findFullFlushMerges
public MergePolicy.MergeSpecification findFullFlushMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext) throws IOException Description copied from class:MergePolicy
Identifies merges that we want to execute (synchronously) on commit. By default, this will returnnatural merges
whose segments are all less than themax segment size for full flushes
.Any merges returned here will make
IndexWriter.commit()
,IndexWriter.prepareCommit()
orIndexWriter.getReader(boolean, boolean)
block until the merges complete or untilLiveIndexWriterConfig.getMaxFullFlushMergeWaitMillis()
has elapsed. This may be used to merge small segments that have just been flushed, reducing the number of segments in the point in time snapshot. If a merge does not complete in the allotted time, it will continue to execute, and eventually finish and apply to future point in time snapshot, but will not be reflected in the current one.If a
MergePolicy.OneMerge
in the returnedMergePolicy.MergeSpecification
includes a segment already included in a registered merge, thenIndexWriter.commit()
orIndexWriter.prepareCommit()
will throw aIllegalStateException
. UseMergePolicy.MergeContext.getMergingSegments()
to determine which segments are currently registered to merge.- Overrides:
findFullFlushMerges
in classFilterMergePolicy
- Parameters:
mergeTrigger
- the event that triggered the merge (COMMIT or GET_READER).segmentInfos
- the total set of segments in the index (while preparing the commit)mergeContext
- the MergeContext to find the merges on, which should be used to determine which segments are already in a registered merge (seeMergePolicy.MergeContext.getMergingSegments()
).- Throws:
IOException
-
findMerges
Description copied from class:MergePolicy
Define the set of merge operations to perform on provided codec readers inIndexWriter.addIndexes(CodecReader...)
.The merge operation is required to convert provided readers into segments that can be added to the writer. This API can be overridden in custom merge policies to control the concurrency for addIndexes. Default implementation creates a single merge operation for all provided readers (lowest concurrency). Creating a merge for each reader, would provide the highest level of concurrency possible with the configured merge scheduler.
- Overrides:
findMerges
in classFilterMergePolicy
- Parameters:
readers
- CodecReader(s) to merge into the main index- Throws:
IOException
-