Package org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class PartitionEventRecordAction

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.action.PartitionEventRecordAction

public class PartitionEventRecordAction
extends java.lang.Object

This class is part of the process for ReadChangeStreamPartitionDoFn SDF. It is responsible for processing PartitionEventRecords. The records will be used to progress the watermark for the current element (partition). Note that because of the returned list of data change record by the connector is not ordered, so the MoveInEvent, MoveOutEvent are not processed by the connector.

Method Summary

Modifier and Type	Method	Description
java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation>	run(PartitionMetadata partition, PartitionEventRecord record, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, RestrictionInterrupter<com.google.cloud.Timestamp> interrupter, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)	This is the main processing function for a PartitionEventRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

run

public java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run(PartitionMetadata partition,
                                                                                       PartitionEventRecord record,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
                                                                                       RestrictionInterrupter<com.google.cloud.Timestamp> interrupter,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)

This is the main processing function for a PartitionEventRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the PartitionEventRecord, so the caller should stop. If an Optional of DoFn.ProcessContinuation.resume() is returned, it means that this function should not attempt to claim further timestamps of the PartitionEventRecord, but instead should commit what it has processed so far.

When processing the PartitionEventRecord the following procedure is applied:

1. We try to claim the partition end record timestamp. If it is not possible, we stop here and return.
2. We update the necessary metrics.
3. We update the watermark to the partition end record timestamp.