com.wcohen.ss.api

Interface Tokenizer

All Known Subinterfaces:

SourcedTokenizer

All Known Implementing Classes:

CharacterTokenizer, NGramTokenizer, SimpleSourcedTokenizer, SimpleTokenizer

public interface Tokenizer

Split a string into tokens.

Method Summary

Modifier and Type	Method and Description
Token	intern(String s) Convert a given string into a token.
int	maxTokenIndex() Return the higest index of any interned token
Iterator	tokenIterator() Return an iterator over interned tokens
Token[]	tokenize(String input) Return tokenized version of a string

Method Detail

tokenize

Token[] tokenize(String input)

Return tokenized version of a string

intern

Token intern(String s)

Convert a given string into a token. The intern function should have these properties: (1) If s1.equals(s2), then intern(s1)==intern(s2). (2) If no string equal to s1 has ever been interned before, then intern(s1).getIndex() will be larger than every previously-assigned index--i.e, token 'indexes' are assigned in increasing order.

tokenIterator

Iterator tokenIterator()

Return an iterator over interned tokens

maxTokenIndex

int maxTokenIndex()

Return the higest index of any interned token