org.opensolaris.opengrok.analysis
Class JFlexTokenizer
java.lang.Object
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.opensolaris.opengrok.analysis.JFlexTokenizer
- Direct Known Subclasses:
- CSymbolTokenizer, CxxSymbolTokenizer, FortranSymbolTokenizer, JavaSymbolTokenizer, LispSymbolTokenizer, PlainFullTokenizer, PlainSymbolTokenizer, ShSymbolTokenizer, TclSymbolTokenizer, TroffFullTokenizer
public abstract class JFlexTokenizer
- extends org.apache.lucene.analysis.Tokenizer
this class was created because of lucene 2.4.1 update which introduced char[] in Tokens instead of String
Created on August 24, 2009
Field Summary |
protected static org.apache.lucene.analysis.Token |
reuseToken
|
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Method Summary |
org.apache.lucene.analysis.Token |
next(org.apache.lucene.analysis.Token preusableToken)
This is a convenience method for having correctly generated classes who reuse Tokens and save gc for lucene summarizer
you MUST consume the returned token to properly get the null value ! |
abstract org.apache.lucene.analysis.Token |
yylex()
|
Methods inherited from class org.apache.lucene.analysis.Tokenizer |
close, reset |
Methods inherited from class org.apache.lucene.analysis.TokenStream |
next, reset |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
reuseToken
protected static final org.apache.lucene.analysis.Token reuseToken
JFlexTokenizer
public JFlexTokenizer()
yylex
public abstract org.apache.lucene.analysis.Token yylex()
throws java.io.IOException
- Throws:
java.io.IOException
next
public final org.apache.lucene.analysis.Token next(org.apache.lucene.analysis.Token preusableToken)
throws java.io.IOException
- This is a convenience method for having correctly generated classes who reuse Tokens and save gc for lucene summarizer
you MUST consume the returned token to properly get the null value !
- Overrides:
next
in class org.apache.lucene.analysis.TokenStream
- Parameters:
preusableToken
-
- Returns:
- null if no more tokens, otherwise a pointer to the modified token
- Throws:
java.io.IOException