org.apache.beam.sdk.io.gcp.bigtable.changestreams.action

Class DetectNewPartitionsAction

java.lang.Object

org.apache.beam.sdk.io.gcp.bigtable.changestreams.action.DetectNewPartitionsAction

@Internal
public class DetectNewPartitionsAction
extends java.lang.Object

This class processes DetectNewPartitionsDoFn.

Constructor Summary

Constructors

Constructor and Description
DetectNewPartitionsAction(ChangeStreamMetrics metrics, MetadataTableDao metadataTableDao, com.google.cloud.Timestamp endTime, GenerateInitialPartitionsAction generateInitialPartitionsAction)

Method Summary

Modifier and Type

Method and Description

org.apache.beam.sdk.transforms.DoFn.ProcessContinuation

run(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionRecord> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer bundleFinalizer, com.google.cloud.Timestamp startTime) Perform the necessary steps to manage initial set of partitions and new partitions.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DetectNewPartitionsAction

public DetectNewPartitionsAction(ChangeStreamMetrics metrics,
                                 MetadataTableDao metadataTableDao,
                                 @Nullable
                                 com.google.cloud.Timestamp endTime,
                                 GenerateInitialPartitionsAction generateInitialPartitionsAction)

Method Detail

run

public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation run(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker,
                                                                   org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionRecord> receiver,
                                                                   org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator,
                                                                   org.apache.beam.sdk.transforms.DoFn.BundleFinalizer bundleFinalizer,
                                                                   com.google.cloud.Timestamp startTime)
                                                            throws java.lang.Exception

Perform the necessary steps to manage initial set of partitions and new partitions. Currently, we set to process new partitions every second.

1. Look up the initial list of partitions to stream if it's the very first run.
2. On rest of the runs, try advancing watermark if needed.
3. Update the metadata table with info about this DoFn.
4. Check if this pipeline has reached the end time. Terminate if it has.
5. Process new partitions and output them.
6. Register callback to clean up processed partitions after bundle has been finalized.

Parameters:

tracker - offset tracker that simply increment by 1 every single run

receiver - output new partitions

watermarkEstimator - update watermark that is a representation of the low watermark of the entire beam pipeline

bundleFinalizer - perform after bundle output actions to clean up metadata table

Returns:

DoFn.ProcessContinuation.resume() with 1-second delay if the stream continues, otherwise DoFn.ProcessContinuation.stop()

Throws:

com.google.protobuf.InvalidProtocolBufferException - if failing to process new partitions

java.lang.Exception