Package org.apache.beam.sdk.io.gcp.bigtable.changestreams.dao

Class MetadataTableAdminDao

  • @Internal
public class MetadataTableAdminDao
extends java.lang.Object
    Data access object for creating and dropping the metadata table.

    The metadata table will be used to keep the state of the entire Beam pipeline as well as splitting and merging partitions.

    Each Beam pipeline will create its own metadata table.

    • Field Detail

      • DEFAULT_METADATA_TABLE_NAME

        public static final java.lang.String DEFAULT_METADATA_TABLE_NAME
        See Also:
        Constant Field Values

      • CF_INITIAL_TOKEN

        public static final java.lang.String CF_INITIAL_TOKEN
        See Also:
        Constant Field Values

      • CF_PARENT_PARTITIONS

        public static final java.lang.String CF_PARENT_PARTITIONS
        See Also:
        Constant Field Values

      • CF_PARENT_LOW_WATERMARKS

        public static final java.lang.String CF_PARENT_LOW_WATERMARKS
        See Also:
        Constant Field Values

      • CF_CONTINUATION_TOKEN

        public static final java.lang.String CF_CONTINUATION_TOKEN
        See Also:
        Constant Field Values

      • CF_MISSING_PARTITIONS

        public static final java.lang.String CF_MISSING_PARTITIONS
        See Also:
        Constant Field Values

      • CF_SHOULD_DELETE

        public static final java.lang.String CF_SHOULD_DELETE
        See Also:
        Constant Field Values

      • QUALIFIER_DEFAULT

        public static final java.lang.String QUALIFIER_DEFAULT
        See Also:
        Constant Field Values

      • COLUMN_FAMILIES

        public static final org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList<java.lang.String> COLUMN_FAMILIES

      • NEW_PARTITION_PREFIX

        public static final com.google.protobuf.ByteString NEW_PARTITION_PREFIX

      • STREAM_PARTITION_PREFIX

        public static final com.google.protobuf.ByteString STREAM_PARTITION_PREFIX

      • DETECT_NEW_PARTITION_SUFFIX

        public static final com.google.protobuf.ByteString DETECT_NEW_PARTITION_SUFFIX

      • CURRENT_METADATA_TABLE_VERSION

        public static final int CURRENT_METADATA_TABLE_VERSION
        See Also:
        Constant Field Values

    • Constructor Detail

      • MetadataTableAdminDao

        public MetadataTableAdminDao​(com.google.cloud.bigtable.admin.v2.BigtableTableAdminClient tableAdminClient,
                             com.google.cloud.bigtable.admin.v2.BigtableInstanceAdminClient instanceAdminClient,
                             java.lang.String changeStreamName,
                             java.lang.String tableId)

    • Method Detail

      • getChangeStreamNamePrefix

        public com.google.protobuf.ByteString getChangeStreamNamePrefix()
        Return the prefix used to identify the rows belonging to this job.
        Returns:
        the prefix used to identify the rows belonging to this job

      • getTableId

        public java.lang.String getTableId()
        Return the metadata table name.
        Returns:
        the metadata table name

      • isAppProfileSingleClusterAndTransactional

        public boolean isAppProfileSingleClusterAndTransactional​(java.lang.String appProfileId)
        Verify the app profile is for single cluster routing with allow single-row transactions enabled. For metadata data operations, the app profile needs to be single cluster routing because it requires read-after-write consistency. Also, the operations depend on single row transactions operations like CheckAndMutateRow.
        Returns:
        true if the app profile is single-cluster and allows single-row transactions, otherwise false

      • doesMetadataTableExist

        public boolean doesMetadataTableExist()
        Returns:
        true if metadata table exists, otherwise false.

      • createMetadataTable

        public boolean createMetadataTable()
        Create the metadata table if it does not exist yet. If the table does exist, verify all the column families exists, if not add those column families. This table only need to be created once per instance. All change streams jobs will use this table. This table is created in the same instance as the table being streamed. While we don't restrict access to the table, manually editing the table can lead to inconsistent beam jobs.
        Returns:
        true if the table was successfully created, otherwise false.

      • cleanUpPrefix

        public void cleanUpPrefix()
        Delete all the metadata rows starting with the change stream name prefix, except for detect new partition row because it signals the existence of a pipeline with the change stream name.