com.clearspring.analytics.stream.cardinality

Class HyperLogLogPlus

java.lang.Object

com.clearspring.analytics.stream.cardinality.HyperLogLogPlus

All Implemented Interfaces:

ICardinality, Serializable

public class HyperLogLogPlus
extends Object
implements ICardinality, Serializable

Implementation of HyperLogLog++ described in

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40671.pdf

Brief HyperLogLog++ Overview

Uses 64 bit hashing instead of 32 Has two representation modes: sparse and normal

'normal' is approximately the same as regular hyperloglog (still uses 64 bits)

'sparse' handles lower cardinality values with a highly accurate but poorly scaling strategy and leverages data compression to compete with 'normal' for as long as possible (sparse has the advantage on accuracy per unit of memory at low cardinality but quickly falls behind).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HyperLogLogPlus.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static int[]	EMPTY_SPARSE

Constructor Summary

Constructors

Constructor and Description
HyperLogLogPlus(int p) This constructor disables the sparse set.
HyperLogLogPlus(int p, int sp) Basic constructor for creating a instance that supports sparse and normal representations.
HyperLogLogPlus(int p, int sp, List<byte[]> deltaByteSet) Constructor to support instances serialized with the legacy sparse encoding scheme.

Method Summary

Methods

Modifier and Type	Method and Description
void	addAll(HyperLogLogPlus other) Add all the elements of the other set to this set.
long	cardinality() Gather the cardinality estimate from this estimator.
boolean	equals(Object obj)
byte[]	getBytes()
int	hashCode()
ICardinality	merge(ICardinality... estimators) Merge this HLL++ with a bunch of others! The power of minions! Most of the logic consists of case analysis about the state of this HLL++ and each one it wants to merge with.
boolean	offer(Object o)
boolean	offerHashed(int hashedInt) Offer the value as a hashed long value
boolean	offerHashed(long hashedLong) Offer the value as a hashed long value
int	sizeof()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

EMPTY_SPARSE

public static final int[] EMPTY_SPARSE

Constructor Detail

HyperLogLogPlus

public HyperLogLogPlus(int p)

This constructor disables the sparse set. If the counter is likely to exceed the sparse set thresholds than using this constructor will help avoid the extra memory pressure created by maintaining the sparse set until that threshold is breached.

Parameters:

p - - the precision value for the normal set

HyperLogLogPlus

public HyperLogLogPlus(int p,
               int sp)

Basic constructor for creating a instance that supports sparse and normal representations. The values of p and sp define the precision of the Normal and Sparse set representations for the data structure. p must be a value between 4 and sp and sp must be less than 32.

Parameters:

p - - the precision value for the normal set

sp - - the precision value for the sparse set

HyperLogLogPlus

public HyperLogLogPlus(int p,
               int sp,
               List<byte[]> deltaByteSet)

Constructor to support instances serialized with the legacy sparse encoding scheme.

Parameters:

p - - the precision value for the normal set

sp - - the precision value for the sparse set

deltaByteSet - - a list of varint byte arrays encoded using a delta encoding scheme

Method Detail

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

offerHashed

public boolean offerHashed(long hashedLong)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedLong - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedLong in the stream

offerHashed

public boolean offerHashed(int hashedInt)

Description copied from interface: ICardinality

Offer the value as a hashed long value

Specified by:

offerHashed in interface ICardinality

Parameters:

hashedInt - - the hash of the item to offer to the estimator

Returns:

false if the value returned by cardinality() is unaffected by the appearance of hashedInt in the stream

offer

public boolean offer(Object o)

Specified by:

offer in interface ICardinality

Parameters:

o - stream element

Returns:

false if the value returned by cardinality() is unaffected by the appearance of o in the stream.

cardinality

public long cardinality()

Gather the cardinality estimate from this estimator.

Has two procedures based on current mode. 'Normal' mode works similar to HLL but has some new bias corrections. 'Sparse' mode is linear counting.

Specified by:

cardinality in interface ICardinality

Returns:

the number of unique elements in the stream or an estimate thereof

sizeof

public int sizeof()

Specified by:

sizeof in interface ICardinality

Returns:

size in bytes needed for serialization

getBytes

public byte[] getBytes()
                throws IOException

Specified by:

getBytes in interface ICardinality

Returns:

Throws:

IOException

addAll

public void addAll(HyperLogLogPlus other)
            throws com.clearspring.analytics.stream.cardinality.HyperLogLogPlus.HyperLogLogPlusMergeException

Add all the elements of the other set to this set.

If possible, the sparse mode is protected. A switch to the normal mode is triggered only if the resulting set exceed the threshold.

This operation does not imply a loss of precision.

Parameters:

other - A compatible Hyperloglog++ instance (same p and sp)

Throws:

CardinalityMergeException - if other is not compatible

com.clearspring.analytics.stream.cardinality.HyperLogLogPlus.HyperLogLogPlusMergeException

merge

public ICardinality merge(ICardinality... estimators)
                   throws CardinalityMergeException

Merge this HLL++ with a bunch of others! The power of minions!

Most of the logic consists of case analysis about the state of this HLL++ and each one it wants to merge with. If either of them is 'normal' mode then the other converts to 'normal' as well. A touching sacrifice. 'Normal's combine just like regular HLL estimators do.

If they happen to be both sparse, then it checks if their combined size would be too large and if so, they get relegated to normal mode anyway. Otherwise, the mergeEstimators function is called, and a new sparse HLL++ is born.

Specified by:

merge in interface ICardinality

Parameters:

estimators - the estimators to merge with this one

Returns:

a new estimator with their combined knowledge

Throws:

CardinalityMergeException