org.apache.beam.sdk.io.gcp.spanner.changestreams.restriction

Class TimestampRangeTracker

java.lang.Object

org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>

org.apache.beam.sdk.io.gcp.spanner.changestreams.restriction.TimestampRangeTracker

All Implemented Interfaces:

org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.HasProgress

Direct Known Subclasses:

DetectNewPartitionsRangeTracker, ReadChangeStreamPartitionRangeTracker

public class TimestampRangeTracker
extends org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>
implements org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.HasProgress

A RestrictionTracker for claiming positions in a TimestampRange in a monotonically increasing fashion.

The smallest position is Timestamp.MIN_VALUE and the largest position is Timestamp.MAX_VALUE - 1 nanosecond.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker

org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.HasProgress, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.IsBounded, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.Progress, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.TruncateResult<RestrictionT>

Field Summary

Fields

Modifier and Type	Field and Description
protected @Nullable com.google.cloud.Timestamp	lastAttemptedPosition
protected @Nullable com.google.cloud.Timestamp	lastClaimedPosition
protected TimestampRange	range
protected java.util.function.Supplier<com.google.cloud.Timestamp>	timeSupplier

Constructor Summary

Constructors

Constructor and Description
TimestampRangeTracker(TimestampRange range)

Method Summary

Modifier and Type	Method and Description
void	checkDone() Checks if the restriction has been processed successfully.
TimestampRange	currentRestriction()
org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.Progress	getProgress() Returns the progress made within the restriction so far.
org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.IsBounded	isBounded()
void	setTimeSupplier(java.util.function.Supplier<com.google.cloud.Timestamp> timeSupplier)
boolean	tryClaim(com.google.cloud.Timestamp position) Attempts to claim the given position.
@Nullable org.apache.beam.sdk.transforms.splittabledofn.SplitResult<TimestampRange>	trySplit(double fractionOfRemainder) Splits the restriction through the following algorithm:

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

range

protected TimestampRange range

lastAttemptedPosition

protected @Nullable com.google.cloud.Timestamp lastAttemptedPosition

lastClaimedPosition

protected @Nullable com.google.cloud.Timestamp lastClaimedPosition

timeSupplier

protected java.util.function.Supplier<com.google.cloud.Timestamp> timeSupplier

Constructor Detail

TimestampRangeTracker

public TimestampRangeTracker(TimestampRange range)

Method Detail

setTimeSupplier

public void setTimeSupplier(java.util.function.Supplier<com.google.cloud.Timestamp> timeSupplier)

tryClaim

public boolean tryClaim(com.google.cloud.Timestamp position)

Attempts to claim the given position. Depending on the position following outcomes are possible:

• If the position is less than or equal to a previous attempted one, an IllegalArgumentException will be thrown
• If the position is less than the restriction range start, an IllegalArgumentException will be thrown
• If the position is greater than or equal to the range end, the position will not be claimed
• If the position is greater or equal to the range start and less than the range end, the position will be claimed

If an error is not thrown, this function will register the position as the lastAttemptedPosition.

Specified by:

tryClaim in class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>

Returns:

true if the position was successfully claimed, false otherwise

trySplit

public @Nullable org.apache.beam.sdk.transforms.splittabledofn.SplitResult<TimestampRange> trySplit(double fractionOfRemainder)

Splits the restriction through the following algorithm:

    currentPosition = lastAttemptedPosition == null ? (from - 1ns) : lastAttemptedPosition
    splitPosition = currentPosition + max(1, (range.to - currentPosition) * fractionOfRemainder)
    primary = [range.from, splitPosition)
    residual = [splitPosition, range.to)
    this.range = primary
 

If the splitPosition is greater than the range.to, null will be returned. For checkpoints the fractionOfRemainder will always be zero.

Specified by:

trySplit in class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>

Returns:

a SplitResult if a split was possible or null if the splitPosition is beyond the end of the range.

checkDone

public void checkDone()
               throws java.lang.IllegalStateException

Checks if the restriction has been processed successfully. If not, throws an IllegalStateException.

The restriction is considered processed successfully if:

• The range is empty (range.from == range.to)
• The lastAttemptedPosition + 1ns >= range.to

The restriction is considered not processed successfully if:

• No position claim was attempted for a non-empty range
• The lastAttemptedPosition + 1ns < range.to

Specified by:

checkDone in class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>

Throws:

java.lang.IllegalStateException

getProgress

public org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.Progress getProgress()

Returns the progress made within the restriction so far. If lastAttemptedPosition is null, the start of the range is used as the completed work; otherwise, lastAttemptedPosition will be used. The time gap between lastAttemptedPosition and now is used as the remaining work. In this way, when the time gap becomes large, we will have more backlog to process and we should add more resources.

Specified by:

getProgress in interface org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.HasProgress

Returns:

work completed and work remaining in seconds.

currentRestriction

public TimestampRange currentRestriction()

Specified by:

currentRestriction in class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>

isBounded

public org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker.IsBounded isBounded()

Specified by:

isBounded in class org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp>