Package org.apache.beam.sdk.io.gcp.bigquery

Class StreamingInserts<DestinationT,ElementT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,ElementT>>,WriteResult>

org.apache.beam.sdk.io.gcp.bigquery.StreamingInserts<DestinationT,ElementT>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class StreamingInserts<DestinationT,ElementT>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,ElementT>>,WriteResult>

PTransform that performs streaming BigQuery write. To increase consistency, it leverages BigQuery's best effort de-dup mechanism.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Constructor Summary

Constructors

Constructor

Description

StreamingInserts(BigQueryIO.Write.CreateDisposition createDisposition, DynamicDestinations<?,DestinationT> dynamicDestinations, org.apache.beam.sdk.coders.Coder<ElementT> elementCoder, org.apache.beam.sdk.transforms.SerializableFunction<ElementT,com.google.api.services.bigquery.model.TableRow> toTableRow, org.apache.beam.sdk.transforms.SerializableFunction<ElementT,com.google.api.services.bigquery.model.TableRow> toFailsafeTableRow)

Constructor.

Method Summary

Modifier and Type	Method	Description
WriteResult	expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,ElementT>> input)
StreamingInserts<DestinationT,ElementT>	withExtendedErrorInfo(boolean extendedErrorInfo)	Specify whether to use extended error info or not.
StreamingInserts<DestinationT,ElementT>	withInsertRetryPolicy(InsertRetryPolicy retryPolicy)	Specify a retry policy for failed inserts.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAdditionalInputs, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

StreamingInserts

public StreamingInserts(BigQueryIO.Write.CreateDisposition createDisposition,
                        DynamicDestinations<?,DestinationT> dynamicDestinations,
                        org.apache.beam.sdk.coders.Coder<ElementT> elementCoder,
                        org.apache.beam.sdk.transforms.SerializableFunction<ElementT,com.google.api.services.bigquery.model.TableRow> toTableRow,
                        org.apache.beam.sdk.transforms.SerializableFunction<ElementT,com.google.api.services.bigquery.model.TableRow> toFailsafeTableRow)

Constructor.

Method Detail

withInsertRetryPolicy

public StreamingInserts<DestinationT,ElementT> withInsertRetryPolicy(InsertRetryPolicy retryPolicy)

Specify a retry policy for failed inserts.

withExtendedErrorInfo

public StreamingInserts<DestinationT,ElementT> withExtendedErrorInfo(boolean extendedErrorInfo)

Specify whether to use extended error info or not.

expand

public WriteResult expand(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,ElementT>> input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<DestinationT,ElementT>>,WriteResult>