Package org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class PartitionEndRecordAction

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.action.PartitionEndRecordAction

public class PartitionEndRecordAction
extends java.lang.Object

This class is part of the process for ReadChangeStreamPartitionDoFn SDF. It is responsible for processing PartitionEndRecords. The records will be used to progress the watermark for the current element (partition). Note that once after this record, the partition will not generate any incomming change records. As the result, the QueryChangeStreamAction will mark the partition reading as finished by itself.

Method Summary

Modifier and Type	Method	Description
java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation>	run(PartitionMetadata partition, PartitionEndRecord record, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, RestrictionInterrupter<com.google.cloud.Timestamp> interrupter, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)	This is the main processing function for a PartitionEndRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

run

public java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run(PartitionMetadata partition,
                                                                                       PartitionEndRecord record,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
                                                                                       RestrictionInterrupter<com.google.cloud.Timestamp> interrupter,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)

This is the main processing function for a PartitionEndRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the PartitionEndRecord, so the caller should stop. If an Optional of DoFn.ProcessContinuation.resume() is returned, it means that this function should not attempt to claim further timestamps of the PartitionEndRecord, but instead should commit what it has processed so far.

When processing the PartitionEndRecord the following procedure is applied:

1. We try to claim the partition end record timestamp. If it is not possible, we stop here and return.
2. We update the necessary metrics.
3. We update the watermark to the partition end record timestamp.