com.clearspring.analytics.stream.cardinality

Class HyperLogLog

java.lang.Object

com.clearspring.analytics.stream.cardinality.HyperLogLog

All Implemented Interfaces:

ICardinality, Serializable

public class HyperLogLog
extends Object
implements ICardinality, Serializable

Java implementation of HyperLogLog (HLL) algorithm from this paper:

http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf

HLL is an improved version of LogLog that is capable of estimating the cardinality of a set with accuracy = 1.04/sqrt(m) where m = 2^b. So we can control accuracy vs space usage by increasing or decreasing b.

The main benefit of using HLL over LL is that it only requires 64% of the space that LL does to get the same accuracy.

This implementation implements a single counter. If a large (millions) number of counters are required you may want to refer to:

http://dsiutils.dsi.unimi.it/

It has a more complex implementation of HLL that supports multiple counters in a single object, drastically reducing the java overhead from creating a large number of objects.

This implementation leveraged a javascript implementation that Yammer has been working on:

https://github.com/yammer/probablyjs

Note that this implementation does not include the long range correction function defined in the original paper. Empirical evidence shows that the correction function causes more harm than good.

Users have different motivations to use different types of hashing functions. Rather than try to keep up with all available hash functions and to remove the concern of causing future binary incompatibilities this class allows clients to offer the value in hashed int or long form. This way clients are free to change their hash function on their own time line. We recommend using Google's Guava Murmur3_128 implementation as it provides good performance and speed when high precision is required. In our tests the 32bit MurmurHash function included in this project is faster and produces better results than the 32 bit murmur3 implementation google provides.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HyperLogLog.Builder
protected static class	HyperLogLog.HyperLogLogMergeException

Constructor Summary

Constructors

Constructor and Description
HyperLogLog(double rsd) Create a new HyperLogLog instance using the specified standard deviation.
HyperLogLog(int log2m) Create a new HyperLogLog instance.
HyperLogLog(int log2m, RegisterSet registerSet) Deprecated.

Method Summary

Methods

Modifier and Type	Method and Description
void	addAll(HyperLogLog other) Add all the elements of the other set to this set.
long	cardinality()
protected static double	getAlphaMM(int p, int m)
byte[]	getBytes()
protected static double	linearCounting(int m, double V)
ICardinality	merge(ICardinality... estimators) Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments.
boolean	offer(Object o)
boolean	offerHashed(int hashedValue) Offer the value as a hashed long value
boolean	offerHashed(long hashedValue) Offer the value as a hashed long value
int	sizeof()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HyperLogLog

public HyperLogLog(double rsd)

Create a new HyperLogLog instance using the specified standard deviation.

Parameters:

rsd - - the relative standard deviation for the counter. smaller values create counters that require more space.

HyperLogLog

public HyperLogLog(int log2m)

Create a new HyperLogLog instance. The log2m parameter defines the accuracy of the counter. The larger the log2m the better the accuracy.

accuracy = 1.04/sqrt(2^log2m)

Parameters:

log2m - - the number of bits to use as the basis for the HLL instance

HyperLogLog

@Deprecated
public HyperLogLog(int log2m,
                      RegisterSet registerSet)

Deprecated.

Creates a new HyperLogLog instance using the given registers. Used for unmarshalling a serialized instance and for merging multiple counters together.

Parameters:

registerSet - - the initial values for the register set

Method Detail

offerHashed

public boolean offerHashed(long hashedValue)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedValue - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream

offerHashed

public boolean offerHashed(int hashedValue)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedValue - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream

offer

public boolean offer(Object o)

Specified by:

offer in interface ICardinality

Parameters:

o - stream element

Returns:

false if the value returned by cardinality() is unaffected by the appearance of o in the stream.

cardinality

public long cardinality()

Specified by:

cardinality in interface ICardinality

Returns:

the number of unique elements in the stream or an estimate thereof

sizeof

public int sizeof()

Specified by:

sizeof in interface ICardinality

Returns:

size in bytes needed for serialization

getBytes

public byte[] getBytes()
                throws IOException

Specified by:

getBytes in interface ICardinality

Returns:

Throws:

IOException

addAll

public void addAll(HyperLogLog other)
            throws CardinalityMergeException

Add all the elements of the other set to this set.

This operation does not imply a loss of precision.

Parameters:

other - A compatible Hyperloglog instance (same log2m)

Throws:

CardinalityMergeException - if other is not compatible

merge

public ICardinality merge(ICardinality... estimators)
                   throws CardinalityMergeException

Description copied from interface: ICardinality

Merges estimators to produce a new estimator for the combined streams of this estimator and those passed as arguments.

Nor this estimator nor the one passed as parameters are modified.

Specified by:

merge in interface ICardinality

Parameters:

estimators - Zero or more compatible estimators

Throws:

CardinalityMergeException - If at least one of the estimators is not compatible with this one

getAlphaMM

protected static double getAlphaMM(int p,
                int m)

linearCounting

protected static double linearCounting(int m,
                    double V)