Package org.apache.beam.sdk.io.gcp.bigquery

Defines transforms for reading and writing from Google BigQuery.

See Also:

BigQueryIO

Interface Summary

Interface	Description
BigQueryOptions	Properties needed when using Google BigQuery with the Apache Beam SDK.
BigQueryServices	An interface for real, mock, or fake implementations of Cloud BigQuery services.
BigQueryServices.BigQueryServerStream<T>	Container for reading data from streaming endpoints.
BigQueryServices.DatasetService	An interface to get, create and delete Cloud BigQuery datasets and tables.
BigQueryServices.JobService	An interface for the Cloud BigQuery load service.
BigQueryServices.StorageClient	An interface representing a client object for making calls to the BigQuery Storage API.
BigQueryServices.StreamAppendClient	An interface for appending records to a Storage API write stream.
BigQueryServices.WriteStreamService	An interface to get, create and flush Cloud BigQuery STORAGE API write streams.
ErrorContainer<T>	ErrorContainer interface.
StreamingInsertsMetrics	Stores and exports metrics for a batch of Streaming Inserts RPCs.
TestBigQuery.PollingAssertion	Interface to implement a polling assertion.
TestBigQueryOptions	TestPipelineOptions for TestBigQuery.

Class Summary

Class	Description
AvroGenericRecordToStorageApiProto	Utility methods for converting Avro GenericRecord objects to dynamic protocol message, for use with the Storage write API.
AvroWriteRequest<T>
BeamRowToStorageApiProto	Utility methods for converting Beam Row objects to dynamic protocol message, for use with the Storage write API.
BigQueryCoderProviderRegistrar	A CoderProviderRegistrar for standard types used with BigQueryIO.
BigQueryDlqProvider
BigQueryExportReadSchemaTransformConfiguration	Configuration for reading from BigQuery.
BigQueryExportReadSchemaTransformConfiguration.Builder
BigQueryExportReadSchemaTransformProvider	An implementation of TypedSchemaTransformProvider for BigQuery read jobs configured using BigQueryExportReadSchemaTransformConfiguration.
BigQueryExportReadSchemaTransformProvider.BigQueryExportSchemaTransform	An implementation of SchemaTransform for BigQuery read jobs configured using BigQueryExportReadSchemaTransformConfiguration.
BigQueryHelpers	A set of helper functions and classes used by BigQueryIO.
BigQueryInsertError	Model definition for BigQueryInsertError.
BigQueryInsertErrorCoder	A Coder that encodes BigQuery BigQueryInsertError objects.
BigQueryIO	PTransforms for reading and writing BigQuery tables.
BigQueryIO.Read	Implementation of BigQueryIO.read().
BigQueryIO.TypedRead<T>	Implementation of BigQueryIO.read(SerializableFunction).
BigQueryIO.Write<T>	Implementation of BigQueryIO.write().
BigQueryIOTranslation
BigQueryIOTranslation.ReadRegistrar
BigQueryIOTranslation.WriteRegistrar
BigQuerySchemaIOProvider	An implementation of SchemaIOProvider for reading and writing to BigQuery with BigQueryIO.
BigQueryServicesImpl	An implementation of BigQueryServices that actually communicates with the cloud BigQuery service.
BigQueryServicesImpl.DatasetServiceImpl
BigQueryServicesImpl.WriteStreamServiceImpl
BigQuerySinkMetrics	Helper class to create perworker metrics for BigQuery Sink stages.
BigQueryStorageApiInsertError
BigQueryStorageApiInsertErrorCoder
BigQueryStorageTableSource<T>	A Source representing reading from a table.
BigQueryUtils	Utility methods for BigQuery related operations.
BigQueryUtils.ConversionOptions	Options for how to convert BigQuery data to Beam data.
BigQueryUtils.ConversionOptions.Builder	Builder for BigQueryUtils.ConversionOptions.
BigQueryUtils.SchemaConversionOptions	Options for how to convert BigQuery schemas to Beam schemas.
BigQueryUtils.SchemaConversionOptions.Builder	Builder for BigQueryUtils.SchemaConversionOptions.
CivilTimeEncoder	Encoder for TIME and DATETIME values, according to civil_time encoding.
CombineAsIterable<T>
CreateTableHelpers
CreateTables<DestinationT,ElementT>	Creates any tables needed before performing streaming writes to the tables.
DynamicDestinations<T,DestinationT>	This class provides the most general way of specifying dynamic BigQuery table destinations.
InsertRetryPolicy	A retry policy for streaming BigQuery inserts.
InsertRetryPolicy.Context	Contains information about a failed insert.
PrepareWrite<InputT,DestinationT extends @NonNull java.lang.Object,OutputT>	Prepare an input PCollection for writing to BigQuery.
ReifyAsIterable<T>	This transforms turns a side input into a singleton PCollection that can be used as the main input for another transform.
RowMutation	A convenience class for applying row updates to BigQuery using BigQueryIO.applyRowMutations().
RowMutation.RowMutationCoder
RowMutationInformation	This class indicates how to apply a row update to BigQuery.
SchemaAndRecord	A wrapper for a GenericRecord and the TableSchema representing the schema of the table (or query) it was generated from.
StorageApiCDC	Constants and variables for CDC support.
StorageApiConvertMessages<DestinationT,ElementT>	A transform that converts messages to protocol buffers in preparation for writing to BigQuery.
StorageApiConvertMessages.ConvertMessagesDoFn<DestinationT extends @NonNull java.lang.Object,ElementT>
StorageApiDynamicDestinationsTableRow<T,DestinationT extends @NonNull java.lang.Object>
StorageApiFlushAndFinalizeDoFn	This DoFn flushes and optionally (if requested) finalizes Storage API streams.
StorageApiLoads<DestinationT,ElementT>	This PTransform manages loads into BigQuery using the Storage API.
StorageApiWritePayload	Class used to wrap elements being sent to the Storage API sinks.
StorageApiWritePayload.Builder
StorageApiWriteRecordsInconsistent<DestinationT,ElementT>	A transform to write sharded records to BigQuery using the Storage API.
StorageApiWritesShardedRecords<DestinationT extends @NonNull java.lang.Object,ElementT>	A transform to write sharded records to BigQuery using the Storage API (Streaming).
StorageApiWriteUnshardedRecords<DestinationT,ElementT>	Write records to the Storage API using a standard batch approach.
StreamingInserts<DestinationT,ElementT>	PTransform that performs streaming BigQuery write.
StreamingInsertsMetrics.NoOpStreamingInsertsMetrics	No-op implementation of StreamingInsertsResults.
StreamingInsertsMetrics.StreamingInsertsMetricsImpl	Metrics of a batch of InsertAll RPCs.
StreamingWriteTables<ElementT>	This transform takes in key-value pairs of TableRow entries and the TableDestination it should be written to.
TableDestination	Encapsulates a BigQuery table destination.
TableDestinationCoder	A coder for TableDestination objects.
TableDestinationCoderV2	A Coder for TableDestination that includes time partitioning information.
TableDestinationCoderV3	A Coder for TableDestination that includes time partitioning and clustering information.
TableRowJsonCoder	A Coder that encodes BigQuery TableRow objects in their native JSON format.
TableRowToStorageApiProto	Utility methods for converting JSON TableRow objects to dynamic protocol message, for use with the Storage write API.
TableSchemaCache	An updatable cache for table schemas.
TableSchemaUpdateUtils	Helper utilities for handling schema-update responses.
TestBigQuery	Test rule which creates a new table with specified schema, with randomized name and exposes few APIs to work with it.
UpdateSchemaDestination<DestinationT>	Update destination schema based on data that is about to be copied into it.
WriteResult	The result of a BigQueryIO.Write transform.

Enum Summary

Enum	Description
BigQueryIO.TypedRead.Method	Determines the method used to read data from BigQuery.
BigQueryIO.TypedRead.QueryPriority	An enumeration type for the priority of a query.
BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings.
BigQueryIO.Write.Method	Determines the method used to insert data in BigQuery.
BigQueryIO.Write.SchemaUpdateOption	An enumeration type for the BigQuery schema update options strings.
BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings.
BigQueryServices.DatasetService.TableMetadataView
BigQuerySinkMetrics.RpcMethod
BigQueryUtils.ConversionOptions.TruncateTimestamps	Controls whether to truncate timestamps to millisecond precision lossily, or to crash when truncation would result.
RowMutationInformation.MutationType

Exception Summary

Exception	Description
BigQuerySchemaRetrievalException	Exception to signal that BigQuery schema retrieval failed.
TableRowToStorageApiProto.SchemaDoesntMatchException
TableRowToStorageApiProto.SchemaTooNarrowException
TableRowToStorageApiProto.SingleValueConversionException