com.wcohen.ss

Class SoftTFIDF

java.lang.Object

com.wcohen.ss.AbstractStringDistance

com.wcohen.ss.AbstractTokenizedStringDistance

com.wcohen.ss.AbstractStatisticalTokenDistance

com.wcohen.ss.TFIDF

com.wcohen.ss.SoftTFIDF

All Implemented Interfaces:

StringDistance, StringDistanceLearner

Direct Known Subclasses:

JaroTFIDF, JaroWinklerTFIDF, MongeElkanTFIDF

public class SoftTFIDF
extends TFIDF

TFIDF-based distance metric, extended to use "soft" token-matching. Specifically, tokens are considered a partial match if they get a good score using an inner string comparator.

On the WHIRL datasets, thresholding JaroWinkler at 0.9 or 0.95 seems to be about right.

Nested Class Summary

Nested classes/interfaces inherited from class com.wcohen.ss.TFIDF

TFIDF.UnitVector

Field Summary

Fields inherited from class com.wcohen.ss.AbstractStatisticalTokenDistance

collectionSize, documentFrequency, totalTokenCount

Fields inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

tokenizer

Constructor Summary

Constructors

Constructor and Description
SoftTFIDF()
SoftTFIDF(StringDistance tokenDistance)
SoftTFIDF(StringDistance tokenDistance, double tokenMatchThreshold)
SoftTFIDF(Tokenizer tokenizer, StringDistance tokenDistance, double tokenMatchThreshold)

Method Summary

Modifier and Type	Method and Description
String	explainScore(StringWrapper s, StringWrapper t) Explain how the distance was computed.
double	getTokenMatchThreshold()
double	score(StringWrapper s, StringWrapper t) This method needs to be implemented by subclasses.
void	setTokenMatchThreshold(double d)
void	setTokenMatchThreshold(Double d)
String	toString()

Methods inherited from class com.wcohen.ss.TFIDF

asUnitVector, getCollectionSize, getDocumentFrequency, getTokens, getVocabularySize, getWeight, main, prepare, setCollectionSize, setDocumentFrequency, setTokenCount

Methods inherited from class com.wcohen.ss.AbstractStatisticalTokenDistance

checkTrainingHasHappened, tokenIterator, train

Methods inherited from class com.wcohen.ss.AbstractTokenizedStringDistance

asBagOfTokens, prepare, setStringWrapperPool

Methods inherited from class com.wcohen.ss.AbstractStringDistance

addExample, doMain, explainScore, getDistance, hasNextQuery, nextQuery, prepare, score, setDistanceInstancePool

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SoftTFIDF

public SoftTFIDF(Tokenizer tokenizer,
                 StringDistance tokenDistance,
                 double tokenMatchThreshold)

SoftTFIDF

public SoftTFIDF(StringDistance tokenDistance,
                 double tokenMatchThreshold)

SoftTFIDF

public SoftTFIDF(StringDistance tokenDistance)

SoftTFIDF

public SoftTFIDF()

Method Detail

setTokenMatchThreshold

public void setTokenMatchThreshold(double d)

setTokenMatchThreshold

public void setTokenMatchThreshold(Double d)

getTokenMatchThreshold

public double getTokenMatchThreshold()

score

public double score(StringWrapper s,
                    StringWrapper t)

Description copied from class: AbstractStringDistance

This method needs to be implemented by subclasses.

Specified by:

score in interface StringDistance

Overrides:

score in class TFIDF

explainScore

public String explainScore(StringWrapper s,
                           StringWrapper t)

Explain how the distance was computed. In the output, the tokens in S and T are listed, and the common tokens are marked with an asterisk.

Specified by:

explainScore in interface StringDistance

Overrides:

explainScore in class TFIDF

toString

public String toString()

Overrides:

toString in class TFIDF