org.apache.flink.runtime.io.network.partition

Class ResultPartition

java.lang.Object

org.apache.flink.runtime.io.network.partition.ResultPartition

All Implemented Interfaces:

AutoCloseable, AvailabilityProvider, ResultPartitionWriter, BufferPoolOwner

Direct Known Subclasses:

ReleaseOnConsumptionResultPartition

public class ResultPartition
extends Object
implements ResultPartitionWriter, BufferPoolOwner

A result partition for data produced by a single task.

This class is the runtime part of a logical IntermediateResultPartition. Essentially, a result partition is a collection of Buffer instances. The buffers are organized in one or more ResultSubpartition instances, which further partition the data depending on the number of consuming tasks and the data DistributionPattern.

Tasks, which consume a result partition have to request one of its subpartitions. The request happens either remotely (see RemoteInputChannel) or locally (see LocalInputChannel)

Life-cycle

The life-cycle of each result partition has three (possibly overlapping) phases:

1. Produce:
2. Consume:
3. Release:

Buffer management

State management

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields

Modifier and Type	Field and Description
protected BufferCompressor	bufferCompressor Used to compress buffer to reduce IO.
protected static org.slf4j.Logger	LOG
int	numTargetKeyGroups
protected ResultPartitionID	partitionId
protected ResultPartitionManager	partitionManager
protected ResultPartitionType	partitionType Type of this partition.
protected ResultSubpartition[]	subpartitions The subpartitions of this partition.

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
ResultPartition(String owningTaskName, ResultPartitionID partitionId, ResultPartitionType partitionType, ResultSubpartition[] subpartitions, int numTargetKeyGroups, ResultPartitionManager partitionManager, BufferCompressor bufferCompressor, org.apache.flink.util.function.FunctionWithException<BufferPoolOwner,BufferPool,IOException> bufferPoolFactory)

Method Summary

Modifier and Type	Method and Description
boolean	addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex) Adds the bufferConsumer to the subpartition with the given index.
void	close()
ResultSubpartitionView	createSubpartitionView(int index, BufferAvailabilityListener availabilityListener) Returns the requested subpartition.
void	fail(Throwable throwable) Fail the production of the partition.
void	finish() Finishes the result partition.
void	flush(int subpartitionIndex) Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.
void	flushAll() Manually trigger consumption from enqueued BufferConsumers in all subpartitions.
ResultSubpartition[]	getAllPartitions()
CompletableFuture<?>	getAvailableFuture()
BufferBuilder	getBufferBuilder() Requests a BufferBuilder from this partition for writing data.
BufferPool	getBufferPool()
Throwable	getFailureCause()
int	getNumberOfQueuedBuffers()
int	getNumberOfSubpartitions()
int	getNumTargetKeyGroups()
String	getOwningTaskName()
ResultPartitionID	getPartitionId()
ResultPartitionType	getPartitionType() Returns the type of this result partition.
boolean	isReleased() Whether this partition is released.
void	release()
void	release(Throwable cause) Releases the result partition.
void	releaseMemory(int toRelease) Releases buffers held by this result partition.
void	setup() Registers a buffer pool with this result partition.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

isApproximatelyAvailable, isAvailable

Field Detail

LOG

protected static final org.slf4j.Logger LOG

partitionId

protected final ResultPartitionID partitionId

partitionType

protected final ResultPartitionType partitionType

Type of this partition. Defines the concrete subpartition implementation to use.

subpartitions

protected final ResultSubpartition[] subpartitions

The subpartitions of this partition. At least one.

partitionManager

protected final ResultPartitionManager partitionManager

numTargetKeyGroups

public final int numTargetKeyGroups

bufferCompressor

@Nullable
protected final BufferCompressor bufferCompressor

Used to compress buffer to reduce IO.

Constructor Detail

ResultPartition

public ResultPartition(String owningTaskName,
                       ResultPartitionID partitionId,
                       ResultPartitionType partitionType,
                       ResultSubpartition[] subpartitions,
                       int numTargetKeyGroups,
                       ResultPartitionManager partitionManager,
                       @Nullable
                       BufferCompressor bufferCompressor,
                       org.apache.flink.util.function.FunctionWithException<BufferPoolOwner,BufferPool,IOException> bufferPoolFactory)

Method Detail

setup

public void setup()
           throws IOException

Registers a buffer pool with this result partition.

There is one pool for each result partition, which is shared by all its sub partitions.

The pool is registered with the partition *after* it as been constructed in order to conform to the life-cycle of task registrations in the TaskExecutor.

Specified by:

setup in interface ResultPartitionWriter

Throws:

IOException

getOwningTaskName

public String getOwningTaskName()

getPartitionId

public ResultPartitionID getPartitionId()

Specified by:

getPartitionId in interface ResultPartitionWriter

getNumberOfSubpartitions

public int getNumberOfSubpartitions()

Specified by:

getNumberOfSubpartitions in interface ResultPartitionWriter

getBufferPool

public BufferPool getBufferPool()

getNumberOfQueuedBuffers

public int getNumberOfQueuedBuffers()

getPartitionType

public ResultPartitionType getPartitionType()

Returns the type of this result partition.

Returns:

result partition type

getBufferBuilder

public BufferBuilder getBufferBuilder()
                               throws IOException,
                                      InterruptedException

Description copied from interface: ResultPartitionWriter

Requests a BufferBuilder from this partition for writing data.

Specified by:

getBufferBuilder in interface ResultPartitionWriter

Throws:

IOException

InterruptedException

addBufferConsumer

public boolean addBufferConsumer(BufferConsumer bufferConsumer,
                                 int subpartitionIndex)
                          throws IOException

Description copied from interface: ResultPartitionWriter

Adds the bufferConsumer to the subpartition with the given index.

This method takes the ownership of the passed bufferConsumer and thus is responsible for releasing it's resources.

To avoid problems with data re-ordering, before adding new BufferConsumer the previously added one the given subpartitionIndex must be marked as BufferConsumer.isFinished().

Specified by:

addBufferConsumer in interface ResultPartitionWriter

Returns:

true if operation succeeded and bufferConsumer was enqueued for consumption.

Throws:

IOException

flushAll

public void flushAll()

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in all subpartitions.

Specified by:

flushAll in interface ResultPartitionWriter

flush

public void flush(int subpartitionIndex)

Description copied from interface: ResultPartitionWriter

Manually trigger consumption from enqueued BufferConsumers in one specified subpartition.

Specified by:

flush in interface ResultPartitionWriter

finish

public void finish()
            throws IOException

Finishes the result partition.

After this operation, it is not possible to add further data to the result partition.

For BLOCKING results, this will trigger the deployment of consuming tasks.

Specified by:

finish in interface ResultPartitionWriter

Throws:

IOException

release

public void release()

release

public void release(Throwable cause)

Releases the result partition.

close

public void close()

Specified by:

close in interface AutoCloseable

fail

public void fail(@Nullable
                 Throwable throwable)

Description copied from interface: ResultPartitionWriter

Fail the production of the partition.

This method propagates non-null failure causes to consumers on a best-effort basis. This call also leads to the release of all resources associated with the partition. Closing of the partition is still needed afterwards if it has not been done before.

Specified by:

fail in interface ResultPartitionWriter

Parameters:

throwable - failure cause

createSubpartitionView

public ResultSubpartitionView createSubpartitionView(int index,
                                                     BufferAvailabilityListener availabilityListener)
                                              throws IOException

Returns the requested subpartition.

Throws:

IOException

getFailureCause

public Throwable getFailureCause()

getNumTargetKeyGroups

public int getNumTargetKeyGroups()

Specified by:

getNumTargetKeyGroups in interface ResultPartitionWriter

releaseMemory

public void releaseMemory(int toRelease)
                   throws IOException

Releases buffers held by this result partition.

This is a callback from the buffer pool, which is registered for result partitions, which are back pressure-free.

Specified by:

releaseMemory in interface BufferPoolOwner

Throws:

IOException

isReleased

public boolean isReleased()

Whether this partition is released.

A partition is released when each subpartition is either consumed and communication is closed by consumer or failed. A partition is also released if task is cancelled.

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface AvailabilityProvider

Returns:

a future that is completed if the respective provider is available.

toString

public String toString()

Overrides:

toString in class Object

getAllPartitions

public ResultSubpartition[] getAllPartitions()