Package org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class DataChangeRecordAction

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.action.DataChangeRecordAction

public class DataChangeRecordAction
extends java.lang.Object

This class is part of the process for ReadChangeStreamPartitionDoFn SDF. It is responsible for processing DataChangeRecords. The records will simply be emitted to the received output receiver.

Constructor Summary

Constructors

Constructor	Description
DataChangeRecordAction(ThroughputEstimator<DataChangeRecord> throughputEstimator)

Method Summary

Modifier and Type	Method	Description
java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation>	run(PartitionMetadata partition, DataChangeRecord record, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, RestrictionInterrupter<com.google.cloud.Timestamp> interrupter, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> outputReceiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)	This is the main processing function for a DataChangeRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataChangeRecordAction

public DataChangeRecordAction(ThroughputEstimator<DataChangeRecord> throughputEstimator)

Parameters:

throughputEstimator - an estimator to calculate local throughput of this action.

Method Detail

run

public java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run(PartitionMetadata partition,
                                                                                       DataChangeRecord record,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
                                                                                       RestrictionInterrupter<com.google.cloud.Timestamp> interrupter,
                                                                                       org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> outputReceiver,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)

This is the main processing function for a DataChangeRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the DataChangeRecord, so the caller should stop.

When processing the DataChangeRecord the following procedure is applied:

1. We try to cliam the data change record commit timestamp. If it is not possible, we stop here and return.
2. We emit the data change record through the DoFn.OutputReceiver.
3. We update the watermark to the data change record commit timestamp.

Parameters:

partition - the current partition being processed

record - the change stream data record received

tracker - the restriction tracker of the ReadChangeStreamPartitionDoFn SDF

interrupter - the restriction interrupter suggesting early termination of the processing

outputReceiver - the output receiver of the ReadChangeStreamPartitionDoFn SDF

watermarkEstimator - the watermark estimator of the ReadChangeStreamPartitionDoFn SDF

Returns:

Optional.empty() if the caller can continue processing more records. A non empty Optional with DoFn.ProcessContinuation.stop() if this function was unable to claim the DataChangeRecord timestamp. A non empty Optional with DoFn.ProcessContinuation.resume() if this function should commit what has already been processed and resume.