org.apache.flink.streaming.runtime.partitioner

Class ForwardForUnspecifiedPartitioner<T>

java.lang.Object

org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardForUnspecifiedPartitioner<T>

Type Parameters:

T - Type of the elements in the Stream

All Implemented Interfaces:

Serializable, org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

@Internal
public class ForwardForUnspecifiedPartitioner<T>
extends ForwardPartitioner<T>

When the parallelism of both upstream and downstream is ExecutionConfig.PARALLELISM_DEFAULT and the edge's partitioner is not specified (partitioner==null), the edge's partitioner will be set to FORWARD by default(See StreamGraph.createActualEdge(java.lang.Integer, java.lang.Integer, int, org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<?>, org.apache.flink.util.OutputTag, org.apache.flink.streaming.api.transformations.StreamExchangeMode, org.apache.flink.runtime.jobgraph.IntermediateDataSetID) method for details). When using the AdaptiveBatchScheduler, this will result in the parallelism of many job vertices is not calculated based on the amount of data but has to align with the parallelism of their upstream vertices due to forward edges, which is contrary to the original intention of the AdaptiveBatchScheduler.

To solve it, we introduce the ForwardForUnspecifiedPartitioner. This partitioner will be set for unspecified edges(partitioner==null), and then the runtime framework will change it to FORWARD/RESCALE after the operator chain creation:

1. Convert to ForwardPartitioner if the partitioner is intra-chain.

2. Convert to RescalePartitioner if the partitioner is inter-chain.

This partitioner should only be used when using AdaptiveBatchScheduler.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

numberOfChannels

Constructor Summary

Constructors

Constructor and Description
ForwardForUnspecifiedPartitioner()

Method Summary

Modifier and Type	Method and Description
StreamPartitioner<T>	copy()
org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper	getDownstreamSubtaskStateMapper() Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.
int	selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

Methods inherited from class org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner

getUpstreamSubtaskStateMapper, isPointwise, toString

Methods inherited from class org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

equals, hashCode, isBroadcast, setup

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ForwardForUnspecifiedPartitioner

public ForwardForUnspecifiedPartitioner()

Method Detail

copy

public StreamPartitioner<T> copy()

Overrides:

copy in class ForwardPartitioner<T>

getDownstreamSubtaskStateMapper

public org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper getDownstreamSubtaskStateMapper()

Description copied from class: StreamPartitioner

Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.

Overrides:

getDownstreamSubtaskStateMapper in class ForwardPartitioner<T>

selectChannel

public int selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

Specified by:

selectChannel in interface org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

Overrides:

selectChannel in class ForwardPartitioner<T>