Package org.apache.beam.sdk.io.gcp.bigquery

Class StorageApiWriteUnshardedRecords<DestinationT,​ElementT>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    public class StorageApiWriteUnshardedRecords<DestinationT,​ElementT>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>>,​org.apache.beam.sdk.values.PCollectionTuple>
    Write records to the Storage API using a standard batch approach. PENDING streams are used, which do not become visible until they are finalized and committed. Each input bundle to the DoFn creates a stream per output table, appends all records in the bundle to the stream, and schedules a finalize/commit operation at the end.
    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.beam.sdk.transforms.PTransform

        annotations, displayData, name, resourceHints

    • Constructor Summary

      Constructors 
      Constructor Description
      StorageApiWriteUnshardedRecords​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,​DestinationT> dynamicDestinations, BigQueryServices bqServices, org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag, @Nullable org.apache.beam.sdk.values.TupleTag<com.google.api.services.bigquery.model.TableRow> successfulRowsTag, java.util.function.Predicate<java.lang.String> successfulRowsPredicate, org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder, org.apache.beam.sdk.coders.Coder<com.google.api.services.bigquery.model.TableRow> successfulRowsCoder, boolean autoUpdateSchema, boolean ignoreUnknownValues, BigQueryIO.Write.CreateDisposition createDisposition, @Nullable java.lang.String kmsKey, boolean usesCdc, com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation defaultMissingValueInterpretation, @Nullable java.util.Map<java.lang.String,​java.lang.String> bigLakeConfiguration)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.beam.sdk.values.PCollectionTuple expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>> input)  

      • Methods inherited from class org.apache.beam.sdk.transforms.PTransform

        addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • StorageApiWriteUnshardedRecords

        public StorageApiWriteUnshardedRecords​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,​DestinationT> dynamicDestinations,
                                       BigQueryServices bqServices,
                                       org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag,
                                       @Nullable org.apache.beam.sdk.values.TupleTag<com.google.api.services.bigquery.model.TableRow> successfulRowsTag,
                                       java.util.function.Predicate<java.lang.String> successfulRowsPredicate,
                                       org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder,
                                       org.apache.beam.sdk.coders.Coder<com.google.api.services.bigquery.model.TableRow> successfulRowsCoder,
                                       boolean autoUpdateSchema,
                                       boolean ignoreUnknownValues,
                                       BigQueryIO.Write.CreateDisposition createDisposition,
                                       @Nullable java.lang.String kmsKey,
                                       boolean usesCdc,
                                       com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation defaultMissingValueInterpretation,
                                       @Nullable java.util.Map<java.lang.String,​java.lang.String> bigLakeConfiguration)

    • Method Detail

      • expand

        public org.apache.beam.sdk.values.PCollectionTuple expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,​StorageApiWritePayload>>,​org.apache.beam.sdk.values.PCollectionTuple>