org.apache.flink.runtime.shuffle

类 NettyShuffleMaster

java.lang.Object

org.apache.flink.runtime.shuffle.NettyShuffleMaster

所有已实现的接口:

AutoCloseable, ShuffleMaster<NettyShuffleDescriptor>

public class NettyShuffleMaster
extends Object
implements ShuffleMaster<NettyShuffleDescriptor>

Default ShuffleMaster for netty and local file based shuffle implementation.

构造器概要

构造器

构造器和说明
NettyShuffleMaster(org.apache.flink.configuration.Configuration conf)

方法概要

限定符和类型	方法和说明
org.apache.flink.configuration.MemorySize	computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc) JM announces network memory requirement from the calculating result of this method.
CompletableFuture<NettyShuffleDescriptor>	registerPartitionWithProducer(org.apache.flink.api.common.JobID jobID, PartitionDescriptor partitionDescriptor, ProducerDescriptor producerDescriptor) Asynchronously register a partition and its producer with the shuffle service.
void	releasePartitionExternally(ShuffleDescriptor shuffleDescriptor) Release any external resources occupied by the given partition.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

从接口继承的方法 org.apache.flink.runtime.shuffle.ShuffleMaster

close, registerJob, start, unregisterJob

构造器详细资料

NettyShuffleMaster

public NettyShuffleMaster(org.apache.flink.configuration.Configuration conf)

方法详细资料

registerPartitionWithProducer

public CompletableFuture<NettyShuffleDescriptor> registerPartitionWithProducer(org.apache.flink.api.common.JobID jobID,
                                                                               PartitionDescriptor partitionDescriptor,
                                                                               ProducerDescriptor producerDescriptor)

从接口复制的说明: ShuffleMaster

Asynchronously register a partition and its producer with the shuffle service.

The returned shuffle descriptor is an internal handle which identifies the partition internally within the shuffle service. The descriptor should provide enough information to read from or write data to the partition.

指定者:

registerPartitionWithProducer 在接口中 ShuffleMaster<NettyShuffleDescriptor>

参数:

jobID - job ID of the corresponding job which registered the partition

partitionDescriptor - general job graph information about the partition

producerDescriptor - general producer information (location, execution id, connection info)

返回:

future with the partition shuffle descriptor used for producer/consumer deployment and their data exchange.

releasePartitionExternally

public void releasePartitionExternally(ShuffleDescriptor shuffleDescriptor)

从接口复制的说明: ShuffleMaster

Release any external resources occupied by the given partition.

This call triggers release of any resources which are occupied by the given partition in the external systems outside of the producer executor. This is mostly relevant for the batch jobs and blocking result partitions. The producer local resources are managed by ShuffleDescriptor.storesLocalResourcesOn() and ShuffleEnvironment.releasePartitionsLocally(Collection).

指定者:

releasePartitionExternally 在接口中 ShuffleMaster<NettyShuffleDescriptor>

参数:

shuffleDescriptor - shuffle descriptor of the result partition to release externally.

computeShuffleMemorySizeForTask

public org.apache.flink.configuration.MemorySize computeShuffleMemorySizeForTask(TaskInputsOutputsDescriptor desc)

JM announces network memory requirement from the calculating result of this method. Please note that the calculating algorithm depends on both I/O details of a vertex and network configuration, e.g. NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL and NettyShuffleEnvironmentOptions.NETWORK_EXTRA_BUFFERS_PER_GATE, which means we should always keep the consistency of configurations between JM, RM and TM in fine-grained resource management, thus to guarantee that the processes of memory announcing and allocating respect each other.

指定者:

computeShuffleMemorySizeForTask 在接口中 ShuffleMaster<NettyShuffleDescriptor>

参数:

desc - describes task inputs and outputs information for shuffle memory calculation.

返回:

shuffle memory size for a task with the given TaskInputsOutputsDescriptor.