com.indeed.mph

Class TableReader<K,V>

java.lang.Object

com.indeed.mph.TableReader<K,V>

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Iterable<com.indeed.util.core.Pair<K,V>>

public class TableReader<K,V>
extends java.lang.Object
implements java.io.Closeable, java.lang.Iterable<com.indeed.util.core.Pair<K,V>>

Class to read from arbitrary mph tables. Mph tables are self-descriptive, so no config is needed. Use SharedTableReader if you need to use the same reader from multiple threads.

Example:

try (final TableReader<KeyClass, ValueClass> reader = TableReader.open(dir)) { final ValueClass value = reader.get(key); }

As a convenience, the main method lets you query existing tables from the command line:

java com.indeed.mph.TableReader [options] table_dir [keys.txt]

where options are

--info: print the meta info --dump: print the key/values of the entire table as TSV --hash: print only the hash value for each key --offset: print only the data offset for each key --get key: print the value for a specific key --random: print the value for a randomly generated long

otherwise prints all values for the associated keys taken from keys.txt (defaulting to stdin). Additional options useful for benchmarking are:

--time: print the time taken to finish all reads --quiet: don't actually print the values --repeat n: repeat the read n times --threads n: duplicate all reads simultaneously among n threads

Author:

alexs

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	TableReader.TableIterator
static class	TableReader.TableStats

Constructor Summary

Constructors

Constructor and Description
TableReader(TableMeta<K,V> meta, byte[] rawData)
TableReader(TableMeta<K,V> meta, com.indeed.util.mmap.MMapBuffer data, com.indeed.util.mmap.MMapBuffer offsets)

Method Summary

Modifier and Type	Method and Description
void	close()
boolean	containsKey(K key)
V	get(K key)
TableConfig<K,V>	getConfig()
V	getForHash(long hash)
long	getHash(K key)
long	getHashOffset(long hash)
K	getKeyForHash(long hash)
com.indeed.util.mmap.MemoryDataInput	getMemoryForHash(long hash)
TableMeta<K,V>	getMeta()
long	getOffset(K key)
long	getSizeInBytes()
TableReader.TableStats	getStats()
long	getTimestamp()
boolean	isDiskBased()
TableReader.TableIterator	iterator()
static void	main(java.lang.String[] args)
static <K,V> TableReader<K,V>	open(java.io.File metaPath)
static <K,V> TableReader<K,V>	open(java.io.File metaPath, java.io.File offsetsPath, java.io.File dataPath)
static <K,V> TableReader<K,V>	open(java.io.File metaPath, java.io.File offsetsPath, java.io.File dataPath, java.lang.Long maxDataHeapUsage) General interface to opening a TableReader.
static <K,V> TableReader<K,V>	open(java.io.File metaPath, java.lang.Long maxDataHeapUsage)
static <K,V> TableReader<K,V>	open(java.lang.String metaPath)
static <K,V> TableReader<K,V>	open(java.lang.String metaPath, java.lang.Long maxDataHeapUsage)
long	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

TableReader

public TableReader(@Nonnull
                   TableMeta<K,V> meta,
                   @Nonnull
                   byte[] rawData)

TableReader

public TableReader(@Nonnull
                   TableMeta<K,V> meta,
                   @Nonnull
                   com.indeed.util.mmap.MMapBuffer data,
                   @Nullable
                   com.indeed.util.mmap.MMapBuffer offsets)

Method Detail

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.io.File metaPath,
                                          @Nullable
                                          java.io.File offsetsPath,
                                          @Nullable
                                          java.io.File dataPath,
                                          @Nullable
                                          java.lang.Long maxDataHeapUsage)
                                   throws java.io.IOException

General interface to opening a TableReader. Only the metaPath is required, variants below are conveniences with default parameters.

Type Parameters:

K - key type

V - value type

Parameters:

metaPath - path to the table's meta-data, either the exact file or the containing directory

offsetsPath - path to the raw offsets if different from the meta-data setting

dataPath - path to the raw serialized data if different from the meta-data setting

maxDataHeapUsage - if specified and positive, raw data less than this size will be stored directly in the heap instead of mmaped

Returns:

a new TableReader on the data

Throws:

java.io.IOException - if unable to open any of the files, or deserialize the metadata

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.io.File metaPath,
                                          @Nullable
                                          java.io.File offsetsPath,
                                          @Nullable
                                          java.io.File dataPath)
                                   throws java.io.IOException

Throws:

java.io.IOException

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.io.File metaPath,
                                          @Nullable
                                          java.lang.Long maxDataHeapUsage)
                                   throws java.io.IOException

Throws:

java.io.IOException

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.io.File metaPath)
                                   throws java.io.IOException

Throws:

java.io.IOException

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.lang.String metaPath)
                                   throws java.io.IOException

Throws:

java.io.IOException

open

public static <K,V> TableReader<K,V> open(@Nonnull
                                          java.lang.String metaPath,
                                          @Nullable
                                          java.lang.Long maxDataHeapUsage)
                                   throws java.io.IOException

Throws:

java.io.IOException

isDiskBased

public boolean isDiskBased()

getMeta

public TableMeta<K,V> getMeta()

getConfig

public TableConfig<K,V> getConfig()

getStats

public TableReader.TableStats getStats()

containsKey

public boolean containsKey(@Nonnull
                           K key)

get

public V get(@Nonnull
             K key)
      throws java.io.IOException

Throws:

java.io.IOException

getMemoryForHash

public com.indeed.util.mmap.MemoryDataInput getMemoryForHash(long hash)
                                                      throws java.io.IOException

Throws:

java.io.IOException

getKeyForHash

public K getKeyForHash(long hash)
                throws java.io.IOException

Throws:

java.io.IOException

getForHash

public V getForHash(long hash)
             throws java.io.IOException

Throws:

java.io.IOException

getHash

public long getHash(@Nonnull
                    K key)

getOffset

public long getOffset(@Nonnull
                      K key)

getHashOffset

public long getHashOffset(long hash)

size

public long size()

getSizeInBytes

public long getSizeInBytes()

getTimestamp

public long getTimestamp()

iterator

public TableReader.TableIterator iterator()

Specified by:

iterator in interface java.lang.Iterable<com.indeed.util.core.Pair<K,V>>

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException,
                        java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException