org.apache.lucene.analysis.minhash

Class MinHashFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.lucene.analysis.minhash.MinHashFilter

All Implemented Interfaces:

Closeable, AutoCloseable

public class MinHashFilter
extends TokenFilter

Generate min hash tokens from an incoming stream of tokens. The incoming tokens would typically be 5 word shingles. The number of hashes used and the number of minimum values for each hash can be set. You could have 1 hash and keep the 100 lowest values or 100 hashes and keep the lowest one for each. Hashes can also be bucketed in ranges over the 128-bit hash space, A 128-bit hash is used internally. 5 word shingles from 10e5 words generate 10e25 combinations So a 64 bit hash would have collisions (1.8e19) When using different hashes 32 bits are used for the hash position leaving scope for 8e28 unique hashes. A single hash will use all 128 bits.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Method Summary

Modifier and Type	Method and Description
void	end()
boolean	incrementToken()
void	reset()

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Method Detail

incrementToken

public final boolean incrementToken()
                             throws IOException

Specified by:

incrementToken in class TokenStream

Throws:

IOException

end

public void end()
         throws IOException

Overrides:

end in class TokenFilter

Throws:

IOException

reset

public void reset()
           throws IOException

Overrides:

reset in class TokenFilter

Throws:

IOException