com.wcohen.ss

Class AbstractSourcedStatisticalTokenDistance

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractSourcedTokenizedStringDistance

com.wcohen.ss.AbstractSourcedStatisticalTokenDistance

All Implemented Interfaces:

StringDistance, StringDistanceLearner

Direct Known Subclasses:

SourcedTFIDF

public abstract class AbstractSourcedStatisticalTokenDistance
extends AbstractSourcedTokenizedStringDistance

Abstract token distance metric that uses frequency statistics.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	collectionSize
protected Map	documentFrequency
protected int	totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
AbstractSourcedStatisticalTokenDistance()
AbstractSourcedStatisticalTokenDistance(SourcedTokenizer tokenizer)

Method Summary

Modifier and Type	Method and Description
protected void	checkTrainingHasHappened(StringWrapper s, StringWrapper t)
int	getDocumentFrequency(Token tok)
void	train(StringWrapperIterator i0) Accumulate statistics on how often each token value occurs

Methods inherited from class com.wcohen.ss.AbstractSourcedTokenizedStringDistance

asBagOfSourcedTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, explainScore, getDistance, hasNextQuery, nextQuery, prepare, prepare, score, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

documentFrequency

protected Map documentFrequency

collectionSize

protected int collectionSize

totalTokenCount

protected int totalTokenCount

Constructor Detail

AbstractSourcedStatisticalTokenDistance

public AbstractSourcedStatisticalTokenDistance(SourcedTokenizer tokenizer)

AbstractSourcedStatisticalTokenDistance

public AbstractSourcedStatisticalTokenDistance()

Method Detail

train

public void train(StringWrapperIterator i0)

Accumulate statistics on how often each token value occurs

Specified by:

train in class AbstractSourcedTokenizedStringDistance

checkTrainingHasHappened

protected void checkTrainingHasHappened(StringWrapper s,
                                        StringWrapper t)

getDocumentFrequency

public int getDocumentFrequency(Token tok)