public class SimpleSourcedTokenizer extends SimpleTokenizer implements SourcedTokenizer
| Modifier and Type | Field and Description |
|---|---|
static SimpleSourcedTokenizer |
DEFAULT_SOURCED_TOKENIZER |
DEFAULT_TOKENIZER| Constructor and Description |
|---|
SimpleSourcedTokenizer(boolean ignorePunctuation,
boolean ignoreCase) |
| Modifier and Type | Method and Description |
|---|---|
static void |
main(String[] argv)
Test routine
|
SourcedToken[] |
sourcedTokenize(String input,
String source)
Return tokenized version of a string.
|
intern, maxTokenIndex, setIgnoreCase, setIgnorePunctuation, tokenIterator, tokenize, toStringclone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitintern, maxTokenIndex, tokenIterator, tokenizepublic static final SimpleSourcedTokenizer DEFAULT_SOURCED_TOKENIZER
public SimpleSourcedTokenizer(boolean ignorePunctuation,
boolean ignoreCase)
public SourcedToken[] sourcedTokenize(String input, String source)
sourcedTokenize in interface SourcedTokenizerpublic static void main(String[] argv)
Copyright © 2016. All rights reserved.