org.apache.flink.runtime.io.network.partition.consumer

Class InputGate

java.lang.Object

org.apache.flink.runtime.io.network.partition.consumer.InputGate

All Implemented Interfaces:

AutoCloseable, AvailabilityProvider, PullingAsyncDataInput<BufferOrEvent>

Direct Known Subclasses:

InputGateWithMetrics, SingleInputGate, UnionInputGate

public abstract class InputGate
extends Object
implements PullingAsyncDataInput<BufferOrEvent>, AutoCloseable

An input gate consumes one or more partitions of a single produced intermediate result.

Each intermediate result is partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

As an example, consider a map-reduce program, where the map operator produces data and the reduce operator consumes the produced data.

 +-----+              +---------------------+              +--------+
 | Map | = produce => | Intermediate Result | <= consume = | Reduce |
 +-----+              +---------------------+              +--------+
 

When deploying such a program in parallel, the intermediate result will be partitioned over its producing parallel subtasks; each of these partitions is furthermore partitioned into one or more subpartitions.

                            Intermediate result
               +-----------------------------------------+
               |                      +----------------+ |              +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 1 | | <=======+=== | Input Gate | Reduce 1 |
 | Map 1 | ==> | | Partition 1 | =|   +----------------+ |         |    +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+    |
               |                      +----------------+ |    |    | Subpartition request
               |                                         |    |    |
               |                      +----------------+ |    |    |
 +-------+     | +-------------+  +=> | Subpartition 1 | | <==+====+
 | Map 2 | ==> | | Partition 2 | =|   +----------------+ |    |         +-----------------------+
 +-------+     | +-------------+  +=> | Subpartition 2 | | <==+======== | Input Gate | Reduce 2 |
               |                      +----------------+ |              +-----------------------+
               +-----------------------------------------+
 

In the above example, two map subtasks produce the intermediate result in parallel, resulting in two partitions (Partition 1 and 2). Each of these partitions is further partitioned into two subpartitions -- one for each parallel reduce subtask. As shown in the Figure, each reduce task will have an input gate attached to it. This will provide its input, which will consist of one subpartition from each partition of the intermediate result.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	InputGate.InputWithData<INPUT,DATA> Simple pojo for INPUT, DATA and moreAvailable.

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AvailabilityProvider.AvailabilityHelper

Field Summary

Fields

Modifier and Type	Field and Description
protected AvailabilityProvider.AvailabilityHelper	availabilityHelper

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
InputGate()

Method Summary

Modifier and Type	Method and Description
CompletableFuture<?>	getAvailableFuture()
abstract Optional<BufferOrEvent>	getNext() Blocking call waiting for next BufferOrEvent.
abstract int	getNumberOfInputChannels()
abstract boolean	isFinished()
abstract Optional<BufferOrEvent>	pollNext() Poll the BufferOrEvent.
abstract void	sendTaskEvent(TaskEvent event)
abstract void	setup() Setup gate, potentially heavy-weight, blocking operation comparing to just creation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

isApproximatelyAvailable, isAvailable

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

availabilityHelper

protected final AvailabilityProvider.AvailabilityHelper availabilityHelper

Constructor Detail

InputGate

public InputGate()

Method Detail

getNumberOfInputChannels

public abstract int getNumberOfInputChannels()

isFinished

public abstract boolean isFinished()

Specified by:

isFinished in interface PullingAsyncDataInput<BufferOrEvent>

Returns:

true if is finished and for example end of input was reached, false otherwise.

getNext

public abstract Optional<BufferOrEvent> getNext()
                                         throws IOException,
                                                InterruptedException

Blocking call waiting for next BufferOrEvent.

Note: It should be guaranteed that the previous returned buffer has been recycled before getting next one.

Returns:

Optional.empty() if isFinished() returns true.

Throws:

IOException

InterruptedException

pollNext

public abstract Optional<BufferOrEvent> pollNext()
                                          throws IOException,
                                                 InterruptedException

Poll the BufferOrEvent.

Note: It should be guaranteed that the previous returned buffer has been recycled before polling next one.

Specified by:

pollNext in interface PullingAsyncDataInput<BufferOrEvent>

Returns:

Optional.empty() if there is no data to return or if isFinished() returns true.

Throws:

IOException

InterruptedException

sendTaskEvent

public abstract void sendTaskEvent(TaskEvent event)
                            throws IOException

Throws:

IOException

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface AvailabilityProvider

Returns:

a future that is completed if there are more records available. If there are more records available immediately, AvailabilityProvider.AVAILABLE should be returned. Previously returned not completed futures should become completed once there are more records available.

setup

public abstract void setup()
                    throws IOException,
                           InterruptedException

Setup gate, potentially heavy-weight, blocking operation comparing to just creation.

Throws:

IOException

InterruptedException