org.apache.flink.runtime.checkpoint

Class StateAssignmentOperation

java.lang.Object

org.apache.flink.runtime.checkpoint.StateAssignmentOperation

@Internal
public class StateAssignmentOperation
extends Object

This class encapsulates the operation of assigning restored state when restoring from a checkpoint.

Constructor Summary

Constructors

Constructor and Description
StateAssignmentOperation(long restoreCheckpointId, Set<ExecutionJobVertex> tasks, Map<OperatorID,OperatorState> operatorStates, boolean allowNonRestoredState)

Method Summary

Modifier and Type	Method and Description
static Map<OperatorInstanceID,List<OperatorStateHandle>>	applyRepartitioner(OperatorID operatorID, OperatorStateRepartitioner opStateRepartitioner, List<List<OperatorStateHandle>> chainOpParallelStates, int oldParallelism, int newParallelism)
static List<List<OperatorStateHandle>>	applyRepartitioner(OperatorStateRepartitioner opStateRepartitioner, List<List<OperatorStateHandle>> chainOpParallelStates, int oldParallelism, int newParallelism) Repartitions the given operator state using the given OperatorStateRepartitioner with respect to the new parallelism.
void	assignStates()
void	checkParallelismPreconditions(List<OperatorState> operatorStates, ExecutionJobVertex executionJobVertex)
static List<KeyGroupRange>	createKeyGroupPartitions(int numberKeyGroups, int parallelism) Groups the available set of key groups into key group partitions.
static List<KeyedStateHandle>	getKeyedStateHandles(Collection<? extends KeyedStateHandle> keyedStateHandles, KeyGroupRange subtaskKeyGroupRange) Determine the subset of KeyGroupsStateHandles with correct key group index for the given subtask KeyGroupRange.
static List<KeyedStateHandle>	getManagedKeyedStateHandles(OperatorState operatorState, KeyGroupRange subtaskKeyGroupRange) Collect managedKeyedStateHandles which have intersection with given KeyGroupRange from operatorState
static List<KeyedStateHandle>	getRawKeyedStateHandles(OperatorState operatorState, KeyGroupRange subtaskKeyGroupRange) Collect rawKeyedStateHandles which have intersection with given KeyGroupRange from operatorState
static OperatorSubtaskState	operatorSubtaskStateFrom(OperatorInstanceID instanceID, Map<OperatorInstanceID,List<OperatorStateHandle>> subManagedOperatorState, Map<OperatorInstanceID,List<OperatorStateHandle>> subRawOperatorState, Map<OperatorInstanceID,List<KeyedStateHandle>> subManagedKeyedState, Map<OperatorInstanceID,List<KeyedStateHandle>> subRawKeyedState)
static void	reDistributePartitionableStates(List<OperatorState> oldOperatorStates, int newParallelism, List<OperatorID> newOperatorIDs, Map<OperatorInstanceID,List<OperatorStateHandle>> newManagedOperatorStates, Map<OperatorInstanceID,List<OperatorStateHandle>> newRawOperatorStates)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StateAssignmentOperation

public StateAssignmentOperation(long restoreCheckpointId,
                                Set<ExecutionJobVertex> tasks,
                                Map<OperatorID,OperatorState> operatorStates,
                                boolean allowNonRestoredState)

Method Detail

assignStates

public void assignStates()

operatorSubtaskStateFrom

public static OperatorSubtaskState operatorSubtaskStateFrom(OperatorInstanceID instanceID,
                                                            Map<OperatorInstanceID,List<OperatorStateHandle>> subManagedOperatorState,
                                                            Map<OperatorInstanceID,List<OperatorStateHandle>> subRawOperatorState,
                                                            Map<OperatorInstanceID,List<KeyedStateHandle>> subManagedKeyedState,
                                                            Map<OperatorInstanceID,List<KeyedStateHandle>> subRawKeyedState)

checkParallelismPreconditions

public void checkParallelismPreconditions(List<OperatorState> operatorStates,
                                          ExecutionJobVertex executionJobVertex)

reDistributePartitionableStates

public static void reDistributePartitionableStates(List<OperatorState> oldOperatorStates,
                                                   int newParallelism,
                                                   List<OperatorID> newOperatorIDs,
                                                   Map<OperatorInstanceID,List<OperatorStateHandle>> newManagedOperatorStates,
                                                   Map<OperatorInstanceID,List<OperatorStateHandle>> newRawOperatorStates)

getManagedKeyedStateHandles

public static List<KeyedStateHandle> getManagedKeyedStateHandles(OperatorState operatorState,
                                                                 KeyGroupRange subtaskKeyGroupRange)

Collect managedKeyedStateHandles which have intersection with given KeyGroupRange from operatorState

Parameters:

operatorState - all state handles of a operator

subtaskKeyGroupRange - the KeyGroupRange of a subtask

Returns:

all managedKeyedStateHandles which have intersection with given KeyGroupRange

getRawKeyedStateHandles

public static List<KeyedStateHandle> getRawKeyedStateHandles(OperatorState operatorState,
                                                             KeyGroupRange subtaskKeyGroupRange)

Collect rawKeyedStateHandles which have intersection with given KeyGroupRange from operatorState

Parameters:

operatorState - all state handles of a operator

subtaskKeyGroupRange - the KeyGroupRange of a subtask

Returns:

all rawKeyedStateHandles which have intersection with given KeyGroupRange

createKeyGroupPartitions

public static List<KeyGroupRange> createKeyGroupPartitions(int numberKeyGroups,
                                                           int parallelism)

Groups the available set of key groups into key group partitions. A key group partition is the set of key groups which is assigned to the same task. Each set of the returned list constitutes a key group partition.

IMPORTANT: The assignment of key groups to partitions has to be in sync with the KeyGroupStreamPartitioner.

Parameters:

numberKeyGroups - Number of available key groups (indexed from 0 to numberKeyGroups - 1)

parallelism - Parallelism to generate the key group partitioning for

Returns:

List of key group partitions

applyRepartitioner

public static Map<OperatorInstanceID,List<OperatorStateHandle>> applyRepartitioner(OperatorID operatorID,
                                                                                   OperatorStateRepartitioner opStateRepartitioner,
                                                                                   List<List<OperatorStateHandle>> chainOpParallelStates,
                                                                                   int oldParallelism,
                                                                                   int newParallelism)

applyRepartitioner

public static List<List<OperatorStateHandle>> applyRepartitioner(OperatorStateRepartitioner opStateRepartitioner,
                                                                 List<List<OperatorStateHandle>> chainOpParallelStates,
                                                                 int oldParallelism,
                                                                 int newParallelism)

Repartitions the given operator state using the given OperatorStateRepartitioner with respect to the new parallelism.

Parameters:

opStateRepartitioner - partitioner to use

chainOpParallelStates - state to repartition

oldParallelism - parallelism with which the state is currently partitioned

newParallelism - parallelism with which the state should be partitioned

Returns:

repartitioned state

getKeyedStateHandles

public static List<KeyedStateHandle> getKeyedStateHandles(Collection<? extends KeyedStateHandle> keyedStateHandles,
                                                          KeyGroupRange subtaskKeyGroupRange)

Determine the subset of KeyGroupsStateHandles with correct key group index for the given subtask KeyGroupRange.

This is publicly visible to be used in tests.