public class SingleInputGate extends InputGate
Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.
+-----+ +---------------------+ +--------+
| Map | = produce => | Intermediate Result | <= consume = | Reduce |
+-----+ +---------------------+ +--------+
When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.
Intermediate result
+-----------------------------------------+
| +----------------+ | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
| Map 1 | ==> | | Partition 1 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+ |
| +----------------+ | | | Subpartition request
| | | |
| +----------------+ | | |
+-------+ | +-------------+ +=> | Subpartition 1 | | <==+====+
| Map 2 | ==> | | Partition 2 | =| +----------------+ | | +-----------------------+
+-------+ | +-------------+ +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
| +----------------+ | +-----------------------+
+-----------------------------------------+
In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask.
InputGate.InputWithData<INPUT,DATA>AvailabilityProvider.AvailabilityHelperavailabilityHelperAVAILABLE| Constructor and Description |
|---|
SingleInputGate(String owningTaskName,
IntermediateDataSetID consumedResultId,
ResultPartitionType consumedPartitionType,
int consumedSubpartitionIndex,
int numberOfInputChannels,
PartitionProducerStateProvider partitionProducerStateProvider,
org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory,
BufferDecompressor bufferDecompressor) |
getAvailableFutureclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitisApproximatelyAvailable, isAvailablepublic SingleInputGate(String owningTaskName, IntermediateDataSetID consumedResultId, ResultPartitionType consumedPartitionType, int consumedSubpartitionIndex, int numberOfInputChannels, PartitionProducerStateProvider partitionProducerStateProvider, org.apache.flink.util.function.SupplierWithException<BufferPool,IOException> bufferPoolFactory, @Nullable BufferDecompressor bufferDecompressor)
public void setup()
throws IOException,
InterruptedException
InputGatesetup in class InputGateIOExceptionInterruptedExceptionpublic int getNumberOfInputChannels()
getNumberOfInputChannels in class InputGatepublic IntermediateDataSetID getConsumedResultId()
public ResultPartitionType getConsumedPartitionType()
public BufferPool getBufferPool()
public int getNumberOfQueuedBuffers()
public CompletableFuture<Void> getCloseFuture()
public void setBufferPool(BufferPool bufferPool)
@VisibleForTesting
public void assignExclusiveSegments()
throws IOException
IOExceptionpublic void setInputChannel(IntermediateResultPartitionID partitionId, InputChannel inputChannel)
public void updateInputChannel(ResourceID localLocation, NettyShuffleDescriptor shuffleDescriptor) throws IOException, InterruptedException
IOExceptionInterruptedExceptionpublic void retriggerPartitionRequest(IntermediateResultPartitionID partitionId) throws IOException
IOExceptionpublic void close()
throws IOException
IOExceptionpublic boolean isFinished()
isFinished in interface PullingAsyncDataInput<BufferOrEvent>isFinished in class InputGatepublic Optional<BufferOrEvent> getNext() throws IOException, InterruptedException
InputGateBufferOrEvent.
Note: It should be guaranteed that the previous returned buffer has been recycled before getting next one.
getNext in class InputGateOptional.empty() if InputGate.isFinished() returns true.IOExceptionInterruptedExceptionpublic Optional<BufferOrEvent> pollNext() throws IOException, InterruptedException
InputGateBufferOrEvent.
Note: It should be guaranteed that the previous returned buffer has been recycled before polling next one.
pollNext in interface PullingAsyncDataInput<BufferOrEvent>pollNext in class InputGateOptional.empty() if there is no data to return or if InputGate.isFinished() returns true.IOExceptionInterruptedExceptionpublic void sendTaskEvent(TaskEvent event) throws IOException
sendTaskEvent in class InputGateIOExceptionpublic Map<IntermediateResultPartitionID,InputChannel> getInputChannels()
Copyright © 2014–2020 The Apache Software Foundation. All rights reserved.