Package org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class PartitionStartRecordAction

  • public class PartitionStartRecordAction
extends java.lang.Object
    This class is part of the process for {@link org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn..ReadChangeStreamPartitionDoFn} SDF. It is responsible for processing PartitionStartRecords. The new partition start records will be stored in the Connector's metadata tables in order to be scheduled for future querying by the org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.dofn.DetectNewPartitionsDoFn SDF.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run​(PartitionMetadata partition, PartitionStartRecord record, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,​com.google.cloud.Timestamp> tracker, RestrictionInterrupter<com.google.cloud.Timestamp> interrupter, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)
      This is the main processing function for a PartitionStartRecord.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • run

        public java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run​(PartitionMetadata partition,
                                                                                       PartitionStartRecord record,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,​com.google.cloud.Timestamp> tracker,
                                                                                       RestrictionInterrupter<com.google.cloud.Timestamp> interrupter,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)
        This is the main processing function for a PartitionStartRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the PartitionStartRecord, so the caller should stop.

        When processing the PartitionStartRecord the following procedure is applied:

        1. We try to claim the partition start record timestamp. If it is not possible, we stop here and return.
        2. We update the watermark to the partition start record timestamp.
        3. For each partition start record, we try to insert them in the metadata tables if they do not exist.
        4. For each partition start record, we increment the corresponding metric.
        Parameters:
        partition - the current partition being processed
        record - the change stream partition start record received
        tracker - the restriction tracker of the com.google.cloud.teleport.spanner.spannerio.changestreams.dofn.ReadChangeStreamPartitionDoFn SDF
        interrupter - the restriction interrupter suggesting early termination of the processing
        watermarkEstimator - the watermark estimator of the com.google.cloud.teleport.spanner.spannerio.changestreams.dofn.ReadChangeStreamPartitionDoFn SDF
        Returns:
        Optional.empty() if the caller can continue processing more records. A non empty Optional with DoFn.ProcessContinuation.stop() if this function was unable to claim the ChildPartitionsRecord timestamp. A non empty Optional with DoFn.ProcessContinuation.resume() if this function should commit what has already been processed and resume.