org.apache.flink.streaming.runtime.partitioner

类 ForwardForConsecutiveHashPartitioner<T>

java.lang.Object

org.apache.flink.streaming.runtime.partitioner.StreamPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner<T>

org.apache.flink.streaming.runtime.partitioner.ForwardForConsecutiveHashPartitioner<T>

类型参数:

T - Type of the elements in the Stream

所有已实现的接口:

Serializable, org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

@Internal
public class ForwardForConsecutiveHashPartitioner<T>
extends ForwardPartitioner<T>

If there are multiple consecutive and the same hash shuffles, SQL planner will change them except the first one to use forward partitioner, so that these operators can be chained to reduce unnecessary shuffles.

 A --[hash]--> B --[hash]--> C
            |
            V
 A --[hash]--> B --[forward]--> C

 

However, sometimes the consecutive hash operators are not chained (e.g. multiple inputs), and this kind of forward partitioners will turn into forward job edges. These forward edges still have the consecutive hash assumption, so that they cannot be changed into rescale/rebalance edges, otherwise it can lead to incorrect results. This prevents the adaptive batch scheduler from determining parallelism for other forward edge downstream job vertices(see FLINK-25046).

To solve it, we introduce the ForwardForConsecutiveHashPartitioner. When SQL planner optimizes the case of multiple consecutive and the same hash shuffles, it should use this partitioner, and then the runtime framework will change it to forward/hash after the operator chain creation.

 A --[hash]--> B --[hash]--> C
            |
            V
 A --[hash]--> B --[ForwardForConsecutiveHash]--> C

 

This partitioner will be converted to following partitioners after the operator chain creation:

1. Be converted to ForwardPartitioner if this partitioner is intra-chain.

2. Be converted to hashPartitioner if this partitioner is inter-chain.

This partitioner should only be used for SQL Batch jobs and when using AdaptiveBatchScheduler.

另请参阅:

序列化表格

字段概要

从类继承的字段 org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

numberOfChannels

构造器概要

构造器

构造器和说明
ForwardForConsecutiveHashPartitioner(StreamPartitioner<T> hashPartitioner) Create a new ForwardForConsecutiveHashPartitioner.

方法概要

限定符和类型	方法和说明
StreamPartitioner<T>	copy()
org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper	getDownstreamSubtaskStateMapper() Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.
StreamPartitioner<T>	getHashPartitioner()
boolean	isPointwise()
int	selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

从类继承的方法 org.apache.flink.streaming.runtime.partitioner.ForwardPartitioner

getUpstreamSubtaskStateMapper, toString

从类继承的方法 org.apache.flink.streaming.runtime.partitioner.StreamPartitioner

equals, hashCode, isBroadcast, setup

从类继承的方法 java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

构造器详细资料

ForwardForConsecutiveHashPartitioner

public ForwardForConsecutiveHashPartitioner(StreamPartitioner<T> hashPartitioner)

Create a new ForwardForConsecutiveHashPartitioner.

参数:

hashPartitioner - the HashPartitioner

方法详细资料

copy

public StreamPartitioner<T> copy()

覆盖:

copy 在类中 ForwardPartitioner<T>

getDownstreamSubtaskStateMapper

public org.apache.flink.runtime.io.network.api.writer.SubtaskStateMapper getDownstreamSubtaskStateMapper()

从类复制的说明: StreamPartitioner

Defines the behavior of this partitioner, when downstream rescaled during recovery of in-flight data.

覆盖:

getDownstreamSubtaskStateMapper 在类中 ForwardPartitioner<T>

isPointwise

public boolean isPointwise()

覆盖:

isPointwise 在类中 ForwardPartitioner<T>

selectChannel

public int selectChannel(org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>> record)

指定者:

selectChannel 在接口中 org.apache.flink.runtime.io.network.api.writer.ChannelSelector<org.apache.flink.runtime.plugable.SerializationDelegate<StreamRecord<T>>>

覆盖:

selectChannel 在类中 ForwardPartitioner<T>

getHashPartitioner

public StreamPartitioner<T> getHashPartitioner()