com.wcohen.ss.tokens

Class SimpleSourcedTokenizer

java.lang.Object

com.wcohen.ss.tokens.SimpleTokenizer

com.wcohen.ss.tokens.SimpleSourcedTokenizer

All Implemented Interfaces:

SourcedTokenizer, Tokenizer

public class SimpleSourcedTokenizer
extends SimpleTokenizer
implements SourcedTokenizer

Simple implementation of a Tokenizer. Tokens are sequences of alphanumerics, optionally including single punctuation characters.

Field Summary

Fields

Modifier and Type	Field and Description
static SimpleSourcedTokenizer	DEFAULT_SOURCED_TOKENIZER

Fields inherited from class com.wcohen.ss.tokens.SimpleTokenizer

DEFAULT_TOKENIZER

Constructor Summary

Constructors

Constructor and Description
SimpleSourcedTokenizer(boolean ignorePunctuation, boolean ignoreCase)

Method Summary

Modifier and Type	Method and Description
static void	main(String[] argv) Test routine
SourcedToken[]	sourcedTokenize(String input, String source) Return tokenized version of a string.

Methods inherited from class com.wcohen.ss.tokens.SimpleTokenizer

intern, maxTokenIndex, setIgnoreCase, setIgnorePunctuation, tokenIterator, tokenize, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.wcohen.ss.api.Tokenizer

intern, maxTokenIndex, tokenIterator, tokenize

Field Detail

DEFAULT_SOURCED_TOKENIZER

public static final SimpleSourcedTokenizer DEFAULT_SOURCED_TOKENIZER

Constructor Detail

SimpleSourcedTokenizer

public SimpleSourcedTokenizer(boolean ignorePunctuation,
                              boolean ignoreCase)

Method Detail

sourcedTokenize

public SourcedToken[] sourcedTokenize(String input,
                                      String source)

Return tokenized version of a string.

Specified by:

sourcedTokenize in interface SourcedTokenizer

main

public static void main(String[] argv)

Test routine