Package org.apache.beam.sdk.io.gcp.bigquery

Class UpdateSchemaDestination<DestinationT>

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<java.lang.Iterable<org.apache.beam.sdk.values.KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,java.lang.Iterable<org.apache.beam.sdk.values.KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>

org.apache.beam.sdk.io.gcp.bigquery.UpdateSchemaDestination<DestinationT>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

public class UpdateSchemaDestination<DestinationT>
extends org.apache.beam.sdk.transforms.DoFn<java.lang.Iterable<org.apache.beam.sdk.values.KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>,java.lang.Iterable<org.apache.beam.sdk.values.KV<TableDestination,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>>>

Update destination schema based on data that is about to be copied into it.

Unlike load and query jobs, BigQuery copy jobs do not support schema field addition or relaxation on the destination table. This DoFn fills that gap by updating the destination table schemas to be compatible with the data coming from the source table so that schemaUpdateOptions are respected regardless of whether data is loaded directly to the destination table or loaded into temporary tables before being copied into the destination.

This transform takes as input a list of KV(destination, WriteTables.Result) and emits a list of KV(TableDestination, WriteTables.Result) where the destination label is parsed and replaced to TableDestination objects.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T extends java.lang.Object>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor

Description

UpdateSchemaDestination(BigQueryServices bqServices, org.apache.beam.sdk.values.PCollectionView<java.lang.String> zeroLoadJobIdPrefixView, org.apache.beam.sdk.options.ValueProvider<java.lang.String> loadJobProjectId, BigQueryIO.Write.WriteDisposition writeDisposition, BigQueryIO.Write.CreateDisposition createDisposition, int maxRetryJobs, java.lang.String kmsKey, java.util.Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions, DynamicDestinations<?,DestinationT> dynamicDestinations)

Method Summary

Modifier and Type	Method	Description
void	finishBundle(org.apache.beam.sdk.transforms.DoFn.FinishBundleContext context)
void	onTeardown()
void	processElement(java.lang.Iterable<org.apache.beam.sdk.values.KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>> element, org.apache.beam.sdk.transforms.DoFn.ProcessContext context, org.apache.beam.sdk.transforms.windowing.BoundedWindow window)
void	startBundle(org.apache.beam.sdk.transforms.DoFn.StartBundleContext c)

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

UpdateSchemaDestination

public UpdateSchemaDestination(BigQueryServices bqServices,
                               org.apache.beam.sdk.values.PCollectionView<java.lang.String> zeroLoadJobIdPrefixView,
                               @Nullable
                               org.apache.beam.sdk.options.ValueProvider<java.lang.String> loadJobProjectId,
                               BigQueryIO.Write.WriteDisposition writeDisposition,
                               BigQueryIO.Write.CreateDisposition createDisposition,
                               int maxRetryJobs,
                               @Nullable
                               java.lang.String kmsKey,
                               java.util.Set<BigQueryIO.Write.SchemaUpdateOption> schemaUpdateOptions,
                               DynamicDestinations<?,DestinationT> dynamicDestinations)

Method Detail

startBundle

@StartBundle
public void startBundle(org.apache.beam.sdk.transforms.DoFn.StartBundleContext c)

processElement

@ProcessElement
public void processElement(@Element
                           java.lang.Iterable<org.apache.beam.sdk.values.KV<DestinationT,org.apache.beam.sdk.io.gcp.bigquery.WriteTables.Result>> element,
                           org.apache.beam.sdk.transforms.DoFn.ProcessContext context,
                           org.apache.beam.sdk.transforms.windowing.BoundedWindow window)
                    throws java.io.IOException

Throws:

java.io.IOException

onTeardown

@Teardown
public void onTeardown()

finishBundle

@FinishBundle
public void finishBundle(org.apache.beam.sdk.transforms.DoFn.FinishBundleContext context)
                  throws java.lang.Exception

Throws:

java.lang.Exception