com.indeed.mph

Class TableWriter

java.lang.Object

com.indeed.mph.TableWriter

public class TableWriter
extends java.lang.Object

Utility class to write mph tables to disk.

The static methods write and writeWithTempStorage are all you need to write an Iterable or Iterator of key-value Pairs to a directory, given a config. The minimum config is just the key serializer - see TableConfig for details. Example:

final TableConfig<Long, Long> config = new TableConfig() .withKeySerializer(new SmartLongSerializer()) .withValueSerializer(new SmartVLongSerializer()); final Set<Pair<Long, Long>> entries = new HashSet<>(); for (long i = 0; i < 20; ++i) { entries.add(new Pair(i, i * i)); } TableWriter.write(new File("squares"), config, entries);

As a convenience, the main method lets you build tables from TSV files:

java com.indeed.recommend.common.platform.mph.TableWriter [options] output_dir input.tsv...

where options are:

--separator: separator to use instead of tab --keySerializer: full class name of the key serializer (default .SmartStringSerializer) --valueSerializer: full class name of the value serializer (default .SmartStringSerializer) --keyStorage: set to IMPLICIT to remove keys from table --offsetStorage: override the default choice --rangeChecking: set to AUTOMATIC to enable tracking of min/max keys --maxHeapUsage: the limit beyond which offsets are mmapped instead of being stored in the heap --signatureWidth: bits per key to use in a bloom filter (required for IMPLICIT keyStorage)

Serializers in the com.indeed.mph.serializers package can be abbreviated with a leading ".", e.g. ".SmartStringSerializer".

The serializer syntax also allows simple expressions of the form Class(args...), notably useful for generic serializers, e.g.:

".SmartListSerializer(.SmartLongSerializer, 20)"

is a serializer for lists of up to 20 Longs.

Author:

alexs

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TableWriter.PairFirstIterable<K,V>
static class	TableWriter.PairFirstIterator<K,V>
static class	TableWriter.PairFirstRangeTrackingIterable<K extends java.lang.Comparable<K>,V>
static class	TableWriter.PairFirstRangeTrackingIterator<K extends java.lang.Comparable<K>,V>
static class	TableWriter.SerializedKeyValueIterable<K,V>
static class	TableWriter.SerializerTransformationStrategy<K>
static class	TableWriter.TsvFileReader<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_TEMP_SHARDS

Method Summary

Modifier and Type	Method and Description
static void	ensureOutputDirectory(java.io.File outputDir)
static void	main(java.lang.String[] args)
static SmartSerializer<java.lang.Object>	parseSerializer(java.lang.String name)
static <K,V> void	write(java.io.File outputDir, TableConfig<K,V> config, java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries) Constructs and writes a minimal perfect hash table to outputDir.
static <K,V> void	write(java.io.File outputDir, TableConfig<K,V> config, java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries, long dataSize) As above, with a pre-computed dataSize.
static <K,V> void	writeWithTempStorage(java.io.File outputDir, TableConfig<K,V> config, java.util.Iterator<com.indeed.util.core.Pair<K,V>> entries) As above, using the outputDir for temp storage.
static <K,V> void	writeWithTempStorage(java.io.File outputDir, TableConfig<K,V> config, java.util.Iterator<com.indeed.util.core.Pair<K,V>> entries, java.io.File tempDir) As above, using a one-time iterator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_TEMP_SHARDS

public static final int MAX_TEMP_SHARDS

See Also:

Constant Field Values

Method Detail

write

public static <K,V> void write(java.io.File outputDir,
                               TableConfig<K,V> config,
                               java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries)
                        throws java.io.IOException

Constructs and writes a minimal perfect hash table to outputDir. The entries may be iterated over multiple times.

Type Parameters:

K - key type

V - value type

Parameters:

outputDir - directory to write the hash table files to

config - a TableConfig specifying at least a key serializer

entries - an iterable of key-value Pairs representing entries in the table

Throws:

java.io.IOException - if unable to write the files or serialize the data

write

public static <K,V> void write(java.io.File outputDir,
                               TableConfig<K,V> config,
                               java.lang.Iterable<com.indeed.util.core.Pair<K,V>> entries,
                               long dataSize)
                        throws java.io.IOException

As above, with a pre-computed dataSize.

Type Parameters:

K - key type

V - value type

Parameters:

outputDir - directory to write the hash table files to

config - a TableConfig specifying at least a key serializer

entries - an iterable of key-value Pairs representing entries in the table

dataSize - the final size of the raw serialized data in the table (omitting keys if using implicit storage, and omitting values if not used)

Throws:

java.io.IOException - if unable to write the files or serialize the data

writeWithTempStorage

public static <K,V> void writeWithTempStorage(java.io.File outputDir,
                                              TableConfig<K,V> config,
                                              java.util.Iterator<com.indeed.util.core.Pair<K,V>> entries,
                                              java.io.File tempDir)
                                       throws java.io.IOException

As above, using a one-time iterator. The entries are written to local temp data, making this suitable for use e.g. when reading from a slow source such as hdfs.

Type Parameters:

K - key type

V - value type

Parameters:

outputDir - directory to write the hash table files to

config - a TableConfig specifying at least a key serializer

entries - an iterable of key-value Pairs representing entries in the table

tempDir - directory to write temporary files to

Throws:

java.io.IOException - if unable to write the files or serialize the data

writeWithTempStorage

public static <K,V> void writeWithTempStorage(java.io.File outputDir,
                                              TableConfig<K,V> config,
                                              java.util.Iterator<com.indeed.util.core.Pair<K,V>> entries)
                                       throws java.io.IOException

As above, using the outputDir for temp storage.

Type Parameters:

K - key type

V - value type

Parameters:

outputDir - directory to write the hash table files to

config - a TableConfig specifying at least a key serializer

entries - an iterable of key-value Pairs representing entries in the table

Throws:

java.io.IOException - if unable to write the files or serialize the data

ensureOutputDirectory

public static void ensureOutputDirectory(java.io.File outputDir)
                                  throws java.io.IOException

Throws:

java.io.IOException

parseSerializer

public static SmartSerializer<java.lang.Object> parseSerializer(java.lang.String name)
                                                         throws java.io.IOException

Throws:

java.io.IOException

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Throws:

java.io.IOException