org.apache.beam.sdk.io.gcp.bigquery

Class StorageApiWriteUnshardedRecords<DestinationT,ElementT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,StorageApiWritePayload>>,org.apache.beam.sdk.values.PCollectionTuple>

org.apache.beam.sdk.io.gcp.bigquery.StorageApiWriteUnshardedRecords<DestinationT,ElementT>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class StorageApiWriteUnshardedRecords<DestinationT,ElementT>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,StorageApiWritePayload>>,org.apache.beam.sdk.values.PCollectionTuple>

Write records to the Storage API using a standard batch approach. PENDING streams are used, which do not become visible until they are finalized and committed. Each input bundle to the DoFn creates a stream per output table, appends all records in the bundle to the stream, and schedules a finalize/commit operation at the end.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
StorageApiWriteUnshardedRecords(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,DestinationT> dynamicDestinations, BigQueryServices bqServices, org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag, org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder, boolean autoUpdateSchema, boolean ignoreUnknownValues)

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.values.PCollectionTuple	expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,StorageApiWritePayload>> input)

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StorageApiWriteUnshardedRecords

public StorageApiWriteUnshardedRecords(org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinations<ElementT,DestinationT> dynamicDestinations,
                                       BigQueryServices bqServices,
                                       org.apache.beam.sdk.values.TupleTag<BigQueryStorageApiInsertError> failedRowsTag,
                                       org.apache.beam.sdk.coders.Coder<BigQueryStorageApiInsertError> failedRowsCoder,
                                       boolean autoUpdateSchema,
                                       boolean ignoreUnknownValues)

Method Detail

expand

public org.apache.beam.sdk.values.PCollectionTuple expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,StorageApiWritePayload>> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,StorageApiWritePayload>>,org.apache.beam.sdk.values.PCollectionTuple>