java.lang.Object
org.apache.lucene.analysis.TokenStreamToAutomaton
- Direct Known Subclasses:
ConcatenateGraphFilter.EscapingTokenStreamToAutomaton
Consumes a TokenStream and creates an
Automaton
where the transition labels are UTF8
bytes (or Unicode code points if unicodeArcs is true) from the TermToBytesRefAttribute
.
Between tokens we insert POS_SEP and for holes we insert HOLE.-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
private static class
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate boolean
static final int
We add this arc to represent a hole.static final int
We create transition between two adjacent tokens.private boolean
private boolean
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate static void
addHoles
(Automaton.Builder builder, RollingBuffer<TokenStreamToAutomaton.Position> positions, int pos) protected BytesRef
changeToken
(BytesRef in) Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph.void
setFinalOffsetGapAsHole
(boolean finalOffsetGapAsHole) If true, any final offset gaps will result in adding a position hole.void
setPreservePositionIncrements
(boolean enablePositionIncrements) Whether to generate holes in the automaton for missing positions,true
by default.void
setUnicodeArcs
(boolean unicodeArcs) Whether to make transition labels Unicode code points instead of UTF8 bytes,false
by defaultPulls the graph (includingPositionLengthAttribute
) from the providedTokenStream
, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
-
Field Details
-
preservePositionIncrements
private boolean preservePositionIncrements -
finalOffsetGapAsHole
private boolean finalOffsetGapAsHole -
unicodeArcs
private boolean unicodeArcs -
POS_SEP
public static final int POS_SEPWe create transition between two adjacent tokens.- See Also:
-
HOLE
public static final int HOLEWe add this arc to represent a hole.- See Also:
-
-
Constructor Details
-
TokenStreamToAutomaton
public TokenStreamToAutomaton()Sole constructor.
-
-
Method Details
-
setPreservePositionIncrements
public void setPreservePositionIncrements(boolean enablePositionIncrements) Whether to generate holes in the automaton for missing positions,true
by default. -
setFinalOffsetGapAsHole
public void setFinalOffsetGapAsHole(boolean finalOffsetGapAsHole) If true, any final offset gaps will result in adding a position hole. -
setUnicodeArcs
public void setUnicodeArcs(boolean unicodeArcs) Whether to make transition labels Unicode code points instead of UTF8 bytes,false
by default -
changeToken
Subclass and implement this if you need to change the token (such as escaping certain bytes) before it's turned into a graph. -
toAutomaton
Pulls the graph (includingPositionLengthAttribute
) from the providedTokenStream
, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.- Throws:
IOException
-
addHoles
private static void addHoles(Automaton.Builder builder, RollingBuffer<TokenStreamToAutomaton.Position> positions, int pos)
-