Uses of Interface
com.wcohen.ss.api.Tokenizer

Packages that use Tokenizer

Package	Description
com.wcohen.ss	This package contains a bunch of approximate string comparators, plus code for performing controlled experiments with this.
com.wcohen.ss.api
com.wcohen.ss.expt
com.wcohen.ss.lookup
com.wcohen.ss.tokens

Uses of Tokenizer in com.wcohen.ss

Fields in com.wcohen.ss declared as Tokenizer

Modifier and Type	Field and Description
protected Tokenizer	AbstractTokenizedStringDistance.tokenizer

Constructors in com.wcohen.ss with parameters of type Tokenizer

Constructor and Description
AbstractSourcedTokenizedStringDistance(Tokenizer tokenizer)
AbstractStatisticalTokenDistance(Tokenizer tokenizer)
AbstractTokenizedStringDistance(Tokenizer tokenizer)
DirichletJS(Tokenizer tokenizer, double pseudoCount)
Jaccard(Tokenizer tokenizer)
JelinekMercerJS(Tokenizer tokenizer, double lambda)
JensenShannonDistance(Tokenizer tokenizer)
Level2(Tokenizer tokenizer, StringDistance tokenDistance)
Mixture(Tokenizer tokenizer)
SoftTFIDF(Tokenizer tokenizer, StringDistance tokenDistance, double tokenMatchThreshold)
SoftTokenFelligiSunter(Tokenizer tokenizer, StringDistance tokenDistance, double tokenMatchThreshold, double mismatchFactor)
TagLink(Tokenizer tokenizer, AbstractStringDistance tokenDistance) TagLink constructor requires a tokenizer and a tokenDistance metric
TFIDF(Tokenizer tokenizer)
TokenFelligiSunter(Tokenizer tokenizer, double mismatchFactor)

Uses of Tokenizer in com.wcohen.ss.api

Subinterfaces of Tokenizer in com.wcohen.ss.api

Modifier and Type	Interface and Description
interface	SourcedTokenizer Split a string into tokens, retaining provinance.

Uses of Tokenizer in com.wcohen.ss.expt

Fields in com.wcohen.ss.expt declared as Tokenizer

Modifier and Type	Field and Description
protected Tokenizer	TokenBlocker.tokenizer
protected Tokenizer	AbbreviationsBlocker.tokenizer

Constructors in com.wcohen.ss.expt with parameters of type Tokenizer

Constructor and Description
AbbreviationsBlocker(Tokenizer tokenizer, double maxFraction)
ClusterTokenBlocker(Tokenizer tokenizer, double maxFraction)
TokenBlocker(Tokenizer tokenizer, double maxFraction)

Uses of Tokenizer in com.wcohen.ss.lookup

Constructors in com.wcohen.ss.lookup with parameters of type Tokenizer

Constructor and Description
SoftDictionary(StringDistanceLearner distanceLearner, Tokenizer tokenizer)
SoftDictionary(Tokenizer tokenizer)
SoftTFIDFDictionary(Tokenizer tokenizer)
SoftTFIDFDictionary(Tokenizer tokenizer, double minTokenSimilarity)
SoftTFIDFDictionary(Tokenizer tokenizer, double minTokenSimilarity, int windowSize, int maxInvertedIndexSize) Create a new SoftTFIDFDictionary.

Uses of Tokenizer in com.wcohen.ss.tokens

Classes in com.wcohen.ss.tokens that implement Tokenizer

Modifier and Type	Class and Description
class	CharacterTokenizer Character tokenizer implementation.
class	NGramTokenizer Wraps another tokenizer, and adds all computes all ngrams of characters from a single token produced by the inner tokenizer.
class	SimpleSourcedTokenizer Simple implementation of a Tokenizer.
class	SimpleTokenizer Simple implementation of a Tokenizer.

Constructors in com.wcohen.ss.tokens with parameters of type Tokenizer

Constructor and Description
NGramTokenizer(int minNGramSize, int maxNGramSize, boolean keepOldTokens, Tokenizer innerTokenizer)