org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryIO.TypedRead<T>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>

org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>, org.apache.beam.sdk.transforms.display.HasDisplayData

Enclosing class:

BigQueryIO

public abstract static class BigQueryIO.TypedRead<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
implements org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>

Implementation of BigQueryIO.read(SerializableFunction).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryIO.TypedRead.Method Determines the method used to read data from BigQuery.
static class	BigQueryIO.TypedRead.QueryPriority An enumeration type for the priority of a query.

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name, resourceHints

Constructor Summary

Constructors

Constructor and Description
TypedRead()

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>	actuateProjectionPushdown(java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.schemas.FieldAccessDescriptor> outputFields)
org.apache.beam.sdk.values.PCollection<T>	expand(org.apache.beam.sdk.values.PBegin input)
BigQueryIO.TypedRead<T>	from(java.lang.String tableSpec) See Read#from(String).
BigQueryIO.TypedRead<T>	from(com.google.api.services.bigquery.model.TableReference table) See Read#from(TableReference).
BigQueryIO.TypedRead<T>	from(org.apache.beam.sdk.options.ValueProvider<java.lang.String> tableSpec) See Read#from(ValueProvider).
BigQueryIO.TypedRead<T>	fromQuery(java.lang.String query) See Read#fromQuery(String).
BigQueryIO.TypedRead<T>	fromQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query) See Read#fromQuery(ValueProvider).
@Nullable com.google.api.services.bigquery.model.TableReference	getTable() See Read#getTable().
@Nullable org.apache.beam.sdk.options.ValueProvider<com.google.api.services.bigquery.model.TableReference>	getTableProvider() See Read#getTableProvider().
void	populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
boolean	supportsProjectionPushdown()
BigQueryIO.TypedRead<T>	useAvroLogicalTypes()
BigQueryIO.TypedRead<T>	usingStandardSql() See Read#usingStandardSql().
void	validate(org.apache.beam.sdk.options.PipelineOptions options)
BigQueryIO.TypedRead<T>	withBeamRowConverters(org.apache.beam.sdk.values.TypeDescriptor<T> typeDescriptor, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn) Sets the functions to convert elements to/from Row objects.
BigQueryIO.TypedRead<T>	withCoder(org.apache.beam.sdk.coders.Coder<T> coder) Sets a Coder for the result of the parse function.
BigQueryIO.TypedRead<T>	withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format) See DataFormat.
BigQueryIO.TypedRead<T>	withKmsKey(java.lang.String kmsKey) For query sources, use this Cloud KMS key to encrypt any temporary tables created.
BigQueryIO.TypedRead<T>	withMethod(BigQueryIO.TypedRead.Method method) See BigQueryIO.TypedRead.Method.
BigQueryIO.TypedRead<T>	withoutResultFlattening() See Read#withoutResultFlattening().
BigQueryIO.TypedRead<T>	withoutValidation() See Read#withoutValidation().
BigQueryIO.TypedRead<T>	withQueryLocation(java.lang.String location) BigQuery geographic location where the query job will be executed.
BigQueryIO.TypedRead<T>	withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority) See BigQueryIO.TypedRead.QueryPriority.
BigQueryIO.TypedRead<T>	withQueryTempDataset(java.lang.String queryTempDatasetRef) Temporary dataset reference when using fromQuery(String).
BigQueryIO.TypedRead<T>	withRowRestriction(java.lang.String rowRestriction) See withRowRestriction(ValueProvider).
BigQueryIO.TypedRead<T>	withRowRestriction(org.apache.beam.sdk.options.ValueProvider<java.lang.String> rowRestriction) Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL.
BigQueryIO.TypedRead<T>	withSelectedFields(java.util.List<java.lang.String> selectedFields) See withSelectedFields(ValueProvider).
BigQueryIO.TypedRead<T>	withSelectedFields(org.apache.beam.sdk.options.ValueProvider<java.util.List<java.lang.String>> selectedFields) Read only the specified fields (columns) from a BigQuery table.
BigQueryIO.TypedRead<T>	withTemplateCompatibility()
BigQueryIO.TypedRead<T>	withTestServices(BigQueryServices testServices)

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TypedRead

public TypedRead()

Method Detail

validate

public void validate(org.apache.beam.sdk.options.PipelineOptions options)

Overrides:

validate in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>

expand

public org.apache.beam.sdk.values.PCollection<T> expand(org.apache.beam.sdk.values.PBegin input)

Specified by:

expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>

populateDisplayData

public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)

Specified by:

populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData

Overrides:

populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>

getTableProvider

public @Nullable org.apache.beam.sdk.options.ValueProvider<com.google.api.services.bigquery.model.TableReference> getTableProvider()

See Read#getTableProvider().

getTable

public @Nullable com.google.api.services.bigquery.model.TableReference getTable()

See Read#getTable().

withCoder

public BigQueryIO.TypedRead<T> withCoder(org.apache.beam.sdk.coders.Coder<T> coder)

Sets a Coder for the result of the parse function. This may be required if a coder can not be inferred automatically.

withKmsKey

public BigQueryIO.TypedRead<T> withKmsKey(java.lang.String kmsKey)

For query sources, use this Cloud KMS key to encrypt any temporary tables created.

withBeamRowConverters

@Experimental(value=SCHEMAS)
public BigQueryIO.TypedRead<T> withBeamRowConverters(org.apache.beam.sdk.values.TypeDescriptor<T> typeDescriptor,
                                                                                  org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn,
                                                                                  org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)

Sets the functions to convert elements to/from Row objects.

Setting these conversion functions is necessary to enable Schema support.

from

public BigQueryIO.TypedRead<T> from(java.lang.String tableSpec)

See Read#from(String).

from

public BigQueryIO.TypedRead<T> from(org.apache.beam.sdk.options.ValueProvider<java.lang.String> tableSpec)

See Read#from(ValueProvider).

fromQuery

public BigQueryIO.TypedRead<T> fromQuery(java.lang.String query)

See Read#fromQuery(String).

fromQuery

public BigQueryIO.TypedRead<T> fromQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query)

See Read#fromQuery(ValueProvider).

from

public BigQueryIO.TypedRead<T> from(com.google.api.services.bigquery.model.TableReference table)

See Read#from(TableReference).

withoutValidation

public BigQueryIO.TypedRead<T> withoutValidation()

See Read#withoutValidation().

withoutResultFlattening

public BigQueryIO.TypedRead<T> withoutResultFlattening()

See Read#withoutResultFlattening().

usingStandardSql

public BigQueryIO.TypedRead<T> usingStandardSql()

See Read#usingStandardSql().

withQueryPriority

public BigQueryIO.TypedRead<T> withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority)

See BigQueryIO.TypedRead.QueryPriority.

withQueryLocation

public BigQueryIO.TypedRead<T> withQueryLocation(java.lang.String location)

BigQuery geographic location where the query job will be executed. If not specified, Beam tries to determine the location by examining the tables referenced by the query. Location must be specified for queries not executed in US or EU, or when you are reading from an authorized view. See BigQuery Jobs: query.

withQueryTempDataset

public BigQueryIO.TypedRead<T> withQueryTempDataset(java.lang.String queryTempDatasetRef)

Temporary dataset reference when using fromQuery(String). When reading from a query, BigQuery will create a temporary dataset and a temporary table to store the results of the query. With this option, you can set an existing dataset to create the temporary table. BigQueryIO will create a temporary table in that dataset, and will remove it once it is not needed. No other tables in the dataset will be modified. If your job does not have permissions to create a new dataset, and you want to use fromQuery(String) (for instance, to read from a view), you should use this option. Remember that the dataset must exist and your job needs permissions to create and remove tables inside that dataset.

withMethod

public BigQueryIO.TypedRead<T> withMethod(BigQueryIO.TypedRead.Method method)

See BigQueryIO.TypedRead.Method.

withFormat

@Experimental(value=SOURCE_SINK)
public BigQueryIO.TypedRead<T> withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format)

See DataFormat.

withSelectedFields

public BigQueryIO.TypedRead<T> withSelectedFields(java.util.List<java.lang.String> selectedFields)

See withSelectedFields(ValueProvider).

withSelectedFields

public BigQueryIO.TypedRead<T> withSelectedFields(org.apache.beam.sdk.options.ValueProvider<java.util.List<java.lang.String>> selectedFields)

Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned.

Requires BigQueryIO.TypedRead.Method.DIRECT_READ. Not compatible with fromQuery(String).

withRowRestriction

public BigQueryIO.TypedRead<T> withRowRestriction(java.lang.String rowRestriction)

See withRowRestriction(ValueProvider).

withRowRestriction

public BigQueryIO.TypedRead<T> withRowRestriction(org.apache.beam.sdk.options.ValueProvider<java.lang.String> rowRestriction)

Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL. If no value is specified, then all rows are returned.

Requires BigQueryIO.TypedRead.Method.DIRECT_READ. Not compatible with fromQuery(String).

withTemplateCompatibility

public BigQueryIO.TypedRead<T> withTemplateCompatibility()

withTestServices

public BigQueryIO.TypedRead<T> withTestServices(BigQueryServices testServices)

useAvroLogicalTypes

public BigQueryIO.TypedRead<T> useAvroLogicalTypes()

supportsProjectionPushdown

public boolean supportsProjectionPushdown()

Specified by:

supportsProjectionPushdown in interface org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>

actuateProjectionPushdown

public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>> actuateProjectionPushdown(java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.schemas.FieldAccessDescriptor> outputFields)

Specified by:

actuateProjectionPushdown in interface org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>