Package org.apache.beam.sdk.io.gcp.spanner

Class SpannerIO.Write

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    SpannerIO
    public abstract static class SpannerIO.Write
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<com.google.cloud.spanner.Mutation>,​SpannerWriteResult>
    A PTransform that writes Mutation objects to Google Cloud Spanner.
    See Also:
    SpannerIO, Serialized Form

    • Constructor Detail

      • Write

        public Write()

    • Method Detail

      • withProjectId

        public SpannerIO.Write withProjectId​(java.lang.String projectId)
        Specifies the Cloud Spanner project.

      • withProjectId

        public SpannerIO.Write withProjectId​(org.apache.beam.sdk.options.ValueProvider<java.lang.String> projectId)
        Specifies the Cloud Spanner project.

      • withInstanceId

        public SpannerIO.Write withInstanceId​(java.lang.String instanceId)
        Specifies the Cloud Spanner instance.

      • withInstanceId

        public SpannerIO.Write withInstanceId​(org.apache.beam.sdk.options.ValueProvider<java.lang.String> instanceId)
        Specifies the Cloud Spanner instance.

      • withDatabaseId

        public SpannerIO.Write withDatabaseId​(java.lang.String databaseId)
        Specifies the Cloud Spanner database.

      • withDatabaseId

        public SpannerIO.Write withDatabaseId​(org.apache.beam.sdk.options.ValueProvider<java.lang.String> databaseId)
        Specifies the Cloud Spanner database.

      • withHost

        public SpannerIO.Write withHost​(org.apache.beam.sdk.options.ValueProvider<java.lang.String> host)
        Specifies the Cloud Spanner host.

      • withHost

        public SpannerIO.Write withHost​(java.lang.String host)
        Specifies the Cloud Spanner host.

      • withEmulatorHost

        public SpannerIO.Write withEmulatorHost​(org.apache.beam.sdk.options.ValueProvider<java.lang.String> emulatorHost)
        Specifies the Cloud Spanner emulator host.

      • withEmulatorHost

        public SpannerIO.Write withEmulatorHost​(java.lang.String emulatorHost)

      • withDialectView

        public SpannerIO.Write withDialectView​(org.apache.beam.sdk.values.PCollectionView<com.google.cloud.spanner.Dialect> dialect)

      • withCommitDeadline

        public SpannerIO.Write withCommitDeadline​(org.joda.time.Duration commitDeadline)
        Specifies the deadline for the Commit API call. Default is 15 secs. DEADLINE_EXCEEDED errors will prompt a backoff/retry until the value of withMaxCumulativeBackoff(Duration) is reached. DEADLINE_EXCEEDED errors are reported with logging and counters.

      • withMaxCommitDelay

        public SpannerIO.Write withMaxCommitDelay​(long millis)
        Specifies max commit delay for the Commit API call for throughput optimized writes. If not set, Spanner might set a small delay if it thinks that will amortize the cost of the writes. For more information about the feature, see documentation

      • withMaxCumulativeBackoff

        public SpannerIO.Write withMaxCumulativeBackoff​(org.joda.time.Duration maxCumulativeBackoff)
        Specifies the maximum cumulative backoff time when retrying after DEADLINE_EXCEEDED errors. Default is 15 mins.

        If the mutations still have not been written after this time, they are treated as a failure, and handled according to the setting of withFailureMode(FailureMode).

      • withBatchSizeBytes

        public SpannerIO.Write withBatchSizeBytes​(long batchSizeBytes)
        Specifies the batch size limit (max number of bytes mutated per batch). Default value is 1MB

      • withMaxNumMutations

        public SpannerIO.Write withMaxNumMutations​(long maxNumMutations)
        Specifies the cell mutation limit (maximum number of mutated cells per batch). Default value is 5000

      • withMaxNumRows

        public SpannerIO.Write withMaxNumRows​(long maxNumRows)
        Specifies the row mutation limit (maximum number of mutated rows per batch). Default value is 1000

      • withSchemaReadySignal

        public SpannerIO.Write withSchemaReadySignal​(org.apache.beam.sdk.values.PCollection<?> signal)
        Specifies an optional input PCollection that can be used as the signal for Wait.OnSignal to indicate when the database schema is ready to be read.

        To be used when the database schema is created by another section of the pipeline, this causes this transform to wait until the signal PCollection has been closed before reading the schema from the database.

        See Also:
        Wait.OnSignal

      • withGroupingFactor

        public SpannerIO.Write withGroupingFactor​(int groupingFactor)
        Specifies the multiple of max mutation (in terms of both bytes per batch and cells per batch) that is used to select a set of mutations to sort by key for batching. This sort uses local memory on the workers, so using large values can cause out of memory errors. Default value is 1000.

      • expand

        public SpannerWriteResult expand​(org.apache.beam.sdk.values.PCollection<com.google.cloud.spanner.Mutation> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<com.google.cloud.spanner.Mutation>,​SpannerWriteResult>

      • populateDisplayData

        public void populateDisplayData​(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
        Specified by:
        populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData
        Overrides:
        populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<com.google.cloud.spanner.Mutation>,​SpannerWriteResult>