org.apache.flink.runtime.operators.hash

Class InMemoryPartition<T>

java.lang.Object

org.apache.flink.runtime.operators.hash.InMemoryPartition<T>

Type Parameters:

T - record type

public class InMemoryPartition<T>
extends Object

In-memory partition with overflow buckets for CompactingHashTable

Field Summary

Fields

Modifier and Type	Field and Description
protected int	nextOverflowBucket
protected int	numOverflowSegments
protected org.apache.flink.core.memory.MemorySegment[]	overflowSegments

Constructor Summary

Constructors

Constructor and Description
InMemoryPartition(org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer, int partitionNumber, ListMemorySegmentSource memSource, int pageSize, int pageSizeInBits) Creates a new partition, in memory, with one buffer.

Method Summary

Modifier and Type	Method and Description
void	allocateSegments(int numberOfSegments) attempts to allocate specified number of segments and should only be used by compaction partition fails silently if not enough segments are available since next compaction could still succeed
long	appendRecord(T record) Inserts the given object into the current buffer.
void	clearAllMemory(List<org.apache.flink.core.memory.MemorySegment> target) releases all of the partition's segments (pages and overflow buckets)
int	getBlockCount()
int	getPartitionNumber() Gets the partition number of this partition.
long	getRecordCount() number of records in partition including garbage
boolean	isCompacted()
void	overwriteRecordAt(long pointer, T record) Deprecated. Don't use this, overwrites record and causes inconsistency or data loss for overwriting everything but records of the exact same size
void	pushDownPages()
T	readRecordAt(long pointer)
T	readRecordAt(long pointer, T reuse)
ArrayList<org.apache.flink.core.memory.MemorySegment>	resetOverflowBuckets() resets overflow bucket counters and returns freed memory and should only be used for resizing
void	resetRecordCounter() sets record counter to zero and should only be used on compaction partition
void	resetRWViews() resets read and write views and should only be used on compaction partition
void	setIsCompacted(boolean compacted) sets compaction status (should only be set true directly after compaction and false when garbage was created)
void	setPartitionNumber(int number) overwrites partition number and should only be used on compaction partition
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

overflowSegments

protected org.apache.flink.core.memory.MemorySegment[] overflowSegments

numOverflowSegments

protected int numOverflowSegments

nextOverflowBucket

protected int nextOverflowBucket

Constructor Detail

InMemoryPartition

public InMemoryPartition(org.apache.flink.api.common.typeutils.TypeSerializer<T> serializer,
                         int partitionNumber,
                         ListMemorySegmentSource memSource,
                         int pageSize,
                         int pageSizeInBits)

Creates a new partition, in memory, with one buffer.

Parameters:

serializer - Serializer for T.

partitionNumber - The number of the partition.

memSource - memory pool

pageSize - segment size in bytes

pageSizeInBits -

Method Detail

getPartitionNumber

public int getPartitionNumber()

Gets the partition number of this partition.

Returns:

This partition's number.

setPartitionNumber

public void setPartitionNumber(int number)

overwrites partition number and should only be used on compaction partition

Parameters:

number - new partition

getBlockCount

public int getBlockCount()

Returns:

number of segments owned by partition

getRecordCount

public long getRecordCount()

number of records in partition including garbage

Returns:

number record count

resetRecordCounter

public void resetRecordCounter()

sets record counter to zero and should only be used on compaction partition

resetRWViews

public void resetRWViews()

resets read and write views and should only be used on compaction partition

pushDownPages

public void pushDownPages()

resetOverflowBuckets

public ArrayList<org.apache.flink.core.memory.MemorySegment> resetOverflowBuckets()

resets overflow bucket counters and returns freed memory and should only be used for resizing

Returns:

freed memory segments

isCompacted

public boolean isCompacted()

Returns:

true if garbage exists in partition

setIsCompacted

public void setIsCompacted(boolean compacted)

sets compaction status (should only be set true directly after compaction and false when garbage was created)

Parameters:

compacted - compaction status

appendRecord

public final long appendRecord(T record)
                        throws IOException

Inserts the given object into the current buffer. This method returns a pointer that can be used to address the written record in this partition.

Parameters:

record - The object to be written to the partition.

Returns:

A pointer to the object in the partition.

Throws:

IOException - Thrown when the write failed.

readRecordAt

public T readRecordAt(long pointer,
                      T reuse)
               throws IOException

Throws:

IOException

readRecordAt

public T readRecordAt(long pointer)
               throws IOException

Throws:

IOException

overwriteRecordAt

@Deprecated
public void overwriteRecordAt(long pointer,
                                          T record)
                                   throws IOException

Deprecated. Don't use this, overwrites record and causes inconsistency or data loss for overwriting everything but records of the exact same size

UNSAFE!! overwrites record causes inconsistency or data loss for overwriting everything but records of the exact same size

Parameters:

pointer - pointer to start of record

record - record to overwrite old one with

Throws:

IOException

clearAllMemory

public void clearAllMemory(List<org.apache.flink.core.memory.MemorySegment> target)

releases all of the partition's segments (pages and overflow buckets)

Parameters:

target - memory pool to release segments to

allocateSegments

public void allocateSegments(int numberOfSegments)

attempts to allocate specified number of segments and should only be used by compaction partition fails silently if not enough segments are available since next compaction could still succeed

Parameters:

numberOfSegments - allocation count

toString

public String toString()

Overrides:

toString in class Object