org.apache.flink.streaming.runtime.io

Class StreamTaskSourceInput<T>

java.lang.Object

org.apache.flink.streaming.runtime.io.StreamTaskSourceInput<T>

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.flink.runtime.io.AvailabilityProvider, org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput, PushingAsyncDataInput<T>, StreamTaskInput<T>

Direct Known Subclasses:

StreamTaskExternallyInducedSourceInput, StreamTaskFinishedOnRestoreSourceInput

@Internal
public class StreamTaskSourceInput<T>
extends Object
implements StreamTaskInput<T>, org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

Implementation of StreamTaskInput that reads data from the SourceOperator and returns the DataInputStatus to indicate whether the source state is available, unavailable or finished.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.runtime.io.PushingAsyncDataInput

PushingAsyncDataInput.DataOutput<T>

Nested classes/interfaces inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

org.apache.flink.runtime.io.AvailabilityProvider.AvailabilityHelper

Field Summary

Fields inherited from interface org.apache.flink.streaming.runtime.io.StreamTaskInput

UNSPECIFIED

Fields inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

AVAILABLE

Constructor Summary

Constructors

Constructor and Description
StreamTaskSourceInput(SourceOperator<T,?> operator, int inputGateIndex, int inputIndex)

Method Summary

Modifier and Type	Method and Description
void	blockConsumption(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)
void	checkpointStarted(org.apache.flink.runtime.io.network.api.CheckpointBarrier barrier) This method is used with unaligned checkpoints to mark the arrival of a first CheckpointBarrier.
void	checkpointStopped(long cancelledCheckpointId)
void	close()
void	convertToPriorityEvent(int channelIndex, int sequenceNumber)
DataInputStatus	emitNext(PushingAsyncDataInput.DataOutput<T> output) Pushes elements to the output from current data input, and returns the input status to indicate whether there are more available data in current input.
CompletableFuture<?>	getAvailableFuture()
List<org.apache.flink.runtime.checkpoint.channel.InputChannelInfo>	getChannelInfos()
int	getInputGateIndex()
int	getInputIndex() Returns the input index of this input.
int	getNumberOfInputChannels()
SourceOperator<T,?>	getOperator()
org.apache.flink.runtime.jobgraph.OperatorID	getOperatorID()
CompletableFuture<Void>	prepareSnapshot(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter channelStateWriter, long checkpointId) Prepares to spill the in-flight input buffers as checkpoint snapshot.
void	resumeConsumption(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.io.AvailabilityProvider

and, isApproximatelyAvailable, isAvailable, or

Constructor Detail

StreamTaskSourceInput

public StreamTaskSourceInput(SourceOperator<T,?> operator,
                             int inputGateIndex,
                             int inputIndex)

Method Detail

emitNext

public DataInputStatus emitNext(PushingAsyncDataInput.DataOutput<T> output)
                         throws Exception

Description copied from interface: PushingAsyncDataInput

Pushes elements to the output from current data input, and returns the input status to indicate whether there are more available data in current input.

This method should be non blocking.

Specified by:

emitNext in interface PushingAsyncDataInput<T>

Throws:

Exception

getAvailableFuture

public CompletableFuture<?> getAvailableFuture()

Specified by:

getAvailableFuture in interface org.apache.flink.runtime.io.AvailabilityProvider

blockConsumption

public void blockConsumption(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)

Specified by:

blockConsumption in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

resumeConsumption

public void resumeConsumption(org.apache.flink.runtime.checkpoint.channel.InputChannelInfo channelInfo)

Specified by:

resumeConsumption in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

getChannelInfos

public List<org.apache.flink.runtime.checkpoint.channel.InputChannelInfo> getChannelInfos()

Specified by:

getChannelInfos in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

getNumberOfInputChannels

public int getNumberOfInputChannels()

Specified by:

getNumberOfInputChannels in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

checkpointStarted

public void checkpointStarted(org.apache.flink.runtime.io.network.api.CheckpointBarrier barrier)

This method is used with unaligned checkpoints to mark the arrival of a first CheckpointBarrier. For chained sources, there is no CheckpointBarrier per se flowing through the job graph. We can assume that an imaginary CheckpointBarrier was produced by the source, at any point of time of our choosing.

We are choosing to interpret it, that CheckpointBarrier for sources was received immediately as soon as we receive either checkpoint start RPC, or CheckpointBarrier from a network input. So that we can checkpoint state of the source and all of the other operators at the same time.

Also we are choosing to block the source, as a best effort optimisation as: - either there is no backpressure and the checkpoint "alignment" will happen very quickly anyway - or there is a backpressure, and it's better to prioritize processing data from the network to speed up checkpointing. From the cluster resource utilisation perspective, by blocking chained source doesn't block any resources from being used, as this task running the source has a backlog of buffered input data waiting to be processed.

However from the correctness point of view, checkpointStarted(CheckpointBarrier) and checkpointStopped(long) methods could be empty no-op.

Specified by:

checkpointStarted in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

checkpointStopped

public void checkpointStopped(long cancelledCheckpointId)

Specified by:

checkpointStopped in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

getInputGateIndex

public int getInputGateIndex()

Specified by:

getInputGateIndex in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

convertToPriorityEvent

public void convertToPriorityEvent(int channelIndex,
                                   int sequenceNumber)
                            throws IOException

Specified by:

convertToPriorityEvent in interface org.apache.flink.runtime.io.network.partition.consumer.CheckpointableInput

Throws:

IOException

getInputIndex

public int getInputIndex()

Description copied from interface: StreamTaskInput

Returns the input index of this input.

Specified by:

getInputIndex in interface StreamTaskInput<T>

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

prepareSnapshot

public CompletableFuture<Void> prepareSnapshot(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter channelStateWriter,
                                               long checkpointId)
                                        throws org.apache.flink.runtime.checkpoint.CheckpointException

Description copied from interface: StreamTaskInput

Prepares to spill the in-flight input buffers as checkpoint snapshot.

Specified by:

prepareSnapshot in interface StreamTaskInput<T>

Throws:

org.apache.flink.runtime.checkpoint.CheckpointException

getOperatorID

public org.apache.flink.runtime.jobgraph.OperatorID getOperatorID()

getOperator

public SourceOperator<T,?> getOperator()