com.indeed.mph

Class MphMap<K,V>

java.lang.Object

com.indeed.mph.MphMap<K,V>

All Implemented Interfaces:

java.io.Serializable, java.util.Map<K,V>

public class MphMap<K,V>
extends java.lang.Object
implements java.util.Map<K,V>, java.io.Serializable

An in-heap mph table implementing the Map interface, with constructors to load from on-disk tables or convert directly from another Map. All of the optional methods (those involving mutation) are unsupported. If keys are stored implicitly, in addition to the normal chance of false positive lookups, the entrySet method will raise an exception. This introduces some computation overhead compared to other Maps but can allow for extremely compact in-heap lookups. As a convenience also implements Serializable, allowing you to store mph tables in resources, use with spark and other distributed processing, etc. at the expense of using more heap and imposing a 2GB limit on the (total, serialized) data size.

Author:

alexs

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface java.util.Map

java.util.Map.Entry<K,V>

Method Summary

Modifier and Type	Method and Description
void	clear()
boolean	containsKey(java.lang.Object o)
boolean	containsValue(java.lang.Object o)
static <K,V> MphMap<K,V>	deserialize(java.io.InputStream in) Static constructor to deserialize an MphMap.
java.util.Set<java.util.Map.Entry<K,V>>	entrySet()
static <K,V> MphMap<K,V>	fromKeyValues(TableConfig<K,V> config, java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries) Static constructor to build a table from key/value pairs, as would be passed to the TableWriter constructor.
static <K,V> MphMap<K,V>	fromMap(TableConfig<K,V> config, java.util.Map<K,V> entries) Static constructor to build a table from another Map.
V	get(java.lang.Object o)
TableConfig<K,V>	getConfig() Returns the config for this mph table, specifying serializers and other info.
V	getForHash(long hash)
long	getHash(K key)
long	getHashOffset(long hash)
K	getKeyForHash(long hash)
com.indeed.util.mmap.MemoryDataInput	getMemoryForHash(long hash)
long	getOffset(K key)
long	getSizeInBytes()
long	getTimestamp()
boolean	isEmpty()
java.util.Set<K>	keySet()
static <K,V> MphMap<K,V>	load(java.io.File metaPath)
static <K,V> MphMap<K,V>	load(java.io.File metaPath, java.io.File offsetsPath, java.io.File dataPath) Static constructor to load a disk-based mph table (as created by TableWriter) into memory.
V	put(K key, V value)
void	putAll(java.util.Map<? extends K,? extends V> m)
V	remove(java.lang.Object key)
int	size()
java.util.Collection<V>	values()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, equals, forEach, getOrDefault, hashCode, merge, putIfAbsent, remove, replace, replace, replaceAll

Method Detail

load

public static <K,V> MphMap<K,V> load(@Nonnull
                                     java.io.File metaPath,
                                     @Nullable
                                     java.io.File offsetsPath,
                                     @Nullable
                                     java.io.File dataPath)
                              throws java.io.IOException

Static constructor to load a disk-based mph table (as created by TableWriter) into memory.

Type Parameters:

K - key type

V - value type

Parameters:

metaPath - path to the table's meta-data, either the exact file or the containing directory

offsetsPath - path to the raw offsets if different from the meta-data setting

dataPath - path to the raw serialized data if different from the meta-data setting

Returns:

a new MphMap holding the data in memory

Throws:

java.io.IOException - if unable to open any of the files, or deserialize the metadata

load

public static <K,V> MphMap<K,V> load(@Nonnull
                                     java.io.File metaPath)
                              throws java.io.IOException

Throws:

java.io.IOException

deserialize

public static <K,V> MphMap<K,V> deserialize(@Nonnull
                                            java.io.InputStream in)
                                     throws java.io.IOException

Static constructor to deserialize an MphMap. This is not the disk-based mph table format.

Type Parameters:

K - key type

V - value type

Parameters:

in - an input stream to read the serialized object from

Returns:

a new MphMap holding the data in memory

Throws:

java.io.IOException - if unable to deserialize

fromMap

public static <K,V> MphMap<K,V> fromMap(@Nonnull
                                        TableConfig<K,V> config,
                                        @Nonnull
                                        java.util.Map<K,V> entries)
                                 throws java.io.IOException

Static constructor to build a table from another Map. This can be a convenient way to compress a Map generated by other means.

Type Parameters:

K - key type

V - value type

Parameters:

config - the TableConfig specifying at least the key/value serializers

entries - the Map to copy from

Returns:

a new MphMap holding the data in memory

Throws:

java.io.IOException - if unable to serialize

fromKeyValues

public static <K,V> MphMap<K,V> fromKeyValues(@Nonnull
                                              TableConfig<K,V> config,
                                              @Nonnull
                                              java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries)
                                       throws java.io.IOException

Static constructor to build a table from key/value pairs, as would be passed to the TableWriter constructor.

Type Parameters:

K - key type

V - value type

Parameters:

config - the TableConfig specifying at least the key/value serializers

entries - an iterable of key/value pairs to covnert to a Map

Returns:

a new MphMap holding the data in memory

Throws:

java.io.IOException - if unable to serialize

getConfig

public TableConfig<K,V> getConfig()

Returns the config for this mph table, specifying serializers and other info.

Returns:

the TableConfig for this mph

containsKey

public boolean containsKey(@Nonnull
                           java.lang.Object o)

Specified by:

containsKey in interface java.util.Map<K,V>

containsValue

public boolean containsValue(java.lang.Object o)

Specified by:

containsValue in interface java.util.Map<K,V>

get

public V get(@Nonnull
             java.lang.Object o)

Specified by:

get in interface java.util.Map<K,V>

getMemoryForHash

public com.indeed.util.mmap.MemoryDataInput getMemoryForHash(long hash)
                                                      throws java.io.IOException

Throws:

java.io.IOException

getKeyForHash

public K getKeyForHash(long hash)
                throws java.io.IOException

Throws:

java.io.IOException

getForHash

public V getForHash(long hash)
             throws java.io.IOException

Throws:

java.io.IOException

getHash

public long getHash(@Nonnull
                    K key)

getOffset

public long getOffset(@Nonnull
                      K key)

getHashOffset

public long getHashOffset(long hash)

size

public int size()

Specified by:

size in interface java.util.Map<K,V>

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface java.util.Map<K,V>

getSizeInBytes

public long getSizeInBytes()

getTimestamp

public long getTimestamp()

entrySet

public java.util.Set<java.util.Map.Entry<K,V>> entrySet()

Specified by:

entrySet in interface java.util.Map<K,V>

keySet

public java.util.Set<K> keySet()

Specified by:

keySet in interface java.util.Map<K,V>

values

public java.util.Collection<V> values()

Specified by:

values in interface java.util.Map<K,V>

clear

public void clear()

Specified by:

clear in interface java.util.Map<K,V>

put

public V put(K key,
             V value)

Specified by:

put in interface java.util.Map<K,V>

putAll

public void putAll(java.util.Map<? extends K,? extends V> m)

Specified by:

putAll in interface java.util.Map<K,V>

remove

public V remove(java.lang.Object key)

Specified by:

remove in interface java.util.Map<K,V>