Package org.apache.beam.sdk.io.gcp.bigquery

Class StorageApiWritesShardedRecords<DestinationT extends @NonNull java.lang.Object,​ElementT>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    public class StorageApiWritesShardedRecords<DestinationT extends @NonNull java.lang.Object,​ElementT>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​java.lang.Iterable<StorageApiWritePayload>>>,​org.apache.beam.sdk.values.PCollectionTuple>
    A transform to write sharded records to BigQuery using the Storage API (Streaming).
    See Also:
    Serialized Form

    • Field Summary

      • Fields inherited from class org.apache.beam.sdk.transforms.PTransform

        annotations, displayData, name, resourceHints

    • Constructor Summary

      Constructors 
      Constructor Description
      StorageApiWritesShardedRecords​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,​DestinationT> dynamicDestinations, BigQueryIO.Write.CreateDisposition createDisposition, java.lang.String kmsKey, BigQueryServices bqServices, org.apache.beam.sdk.coders.Coder<DestinationT> destinationCoder, org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder, org.apache.beam.sdk.coders.Coder<com.google.api.services.bigquery.model.TableRow> successfulRowsCoder, org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag, @Nullable org.apache.beam.sdk.values.TupleTag<com.google.api.services.bigquery.model.TableRow> successfulRowsTag, java.util.function.Predicate<java.lang.String> successfulRowsPredicate, boolean autoUpdateSchema, boolean ignoreUnknownValues, com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation defaultMissingValueInterpretation, @Nullable java.util.Map<java.lang.String,​java.lang.String> bigLakeConfiguration)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.beam.sdk.values.PCollectionTuple expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​java.lang.Iterable<StorageApiWritePayload>>> input)  

      • Methods inherited from class org.apache.beam.sdk.transforms.PTransform

        addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • StorageApiWritesShardedRecords

        public StorageApiWritesShardedRecords​(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,​DestinationT> dynamicDestinations,
                                      BigQueryIO.Write.CreateDisposition createDisposition,
                                      java.lang.String kmsKey,
                                      BigQueryServices bqServices,
                                      org.apache.beam.sdk.coders.Coder<DestinationT> destinationCoder,
                                      org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder,
                                      org.apache.beam.sdk.coders.Coder<com.google.api.services.bigquery.model.TableRow> successfulRowsCoder,
                                      org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag,
                                      @Nullable org.apache.beam.sdk.values.TupleTag<com.google.api.services.bigquery.model.TableRow> successfulRowsTag,
                                      java.util.function.Predicate<java.lang.String> successfulRowsPredicate,
                                      boolean autoUpdateSchema,
                                      boolean ignoreUnknownValues,
                                      com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation defaultMissingValueInterpretation,
                                      @Nullable java.util.Map<java.lang.String,​java.lang.String> bigLakeConfiguration)

    • Method Detail

      • expand

        public org.apache.beam.sdk.values.PCollectionTuple expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT>,​java.lang.Iterable<StorageApiWritePayload>>> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<org.apache.beam.sdk.util.ShardedKey<DestinationT extends @NonNull java.lang.Object>,​java.lang.Iterable<StorageApiWritePayload>>>,​org.apache.beam.sdk.values.PCollectionTuple>