Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQueryIO.TypedRead<T>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>>, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    BigQueryIO
    public abstract static class BigQueryIO.TypedRead<T>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>
implements org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>>
    Implementation of BigQueryIO.read(SerializableFunction).
    See Also:
    Serialized Form

    • Constructor Detail

      • TypedRead

        public TypedRead()

    • Method Detail

      • validate

        public void validate​(org.apache.beam.sdk.options.PipelineOptions options)
        Overrides:
        validate in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>

      • expand

        public org.apache.beam.sdk.values.PCollection<T> expand​(org.apache.beam.sdk.values.PBegin input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>

      • readSource

        public static <T> void readSource​(org.apache.beam.sdk.options.PipelineOptions options,
                                  org.apache.beam.sdk.values.TupleTag<T> rowTag,
                                  org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver outputReceiver,
                                  org.apache.beam.sdk.io.BoundedSource<T> streamSource,
                                  org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ErrorHandlingParseFn<T> errorHandlingParseFn,
                                  org.apache.beam.sdk.transforms.errorhandling.BadRecordRouter badRecordRouter)
                           throws java.lang.Exception
        Throws:
        java.lang.Exception

      • populateDisplayData

        public void populateDisplayData​(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
        Specified by:
        populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData
        Overrides:
        populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>>

      • getTableProvider

        public @Nullable org.apache.beam.sdk.options.ValueProvider<com.google.api.services.bigquery.model.TableReference> getTableProvider()
        See BigQueryIO.Read.getTableProvider().

      • withCoder

        public BigQueryIO.TypedRead<T> withCoder​(org.apache.beam.sdk.coders.Coder<T> coder)
        Sets a Coder for the result of the parse function. This may be required if a coder can not be inferred automatically.

      • withKmsKey

        public BigQueryIO.TypedRead<T> withKmsKey​(java.lang.String kmsKey)
        For query sources, use this Cloud KMS key to encrypt any temporary tables created.

      • withBeamRowConverters

        public BigQueryIO.TypedRead<T> withBeamRowConverters​(org.apache.beam.sdk.values.TypeDescriptor<T> typeDescriptor,
                                                     org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn,
                                                     org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)
        Sets the functions to convert elements to/from Row objects.

        Setting these conversion functions is necessary to enable Schema support.

      • withQueryLocation

        public BigQueryIO.TypedRead<T> withQueryLocation​(java.lang.String location)
        BigQuery geographic location where the query job will be executed. If not specified, Beam tries to determine the location by examining the tables referenced by the query. Location must be specified for queries not executed in US or EU, or when you are reading from an authorized view. See BigQuery Jobs: query.

      • withQueryTempDataset

        public BigQueryIO.TypedRead<T> withQueryTempDataset​(java.lang.String queryTempDatasetRef)
        Temporary dataset reference when using fromQuery(String). When reading from a query, BigQuery will create a temporary dataset and a temporary table to store the results of the query. With this option, you can set an existing dataset to create the temporary table. BigQueryIO will create a temporary table in that dataset, and will remove it once it is not needed. No other tables in the dataset will be modified. If your job does not have permissions to create a new dataset, and you want to use fromQuery(String) (for instance, to read from a view), you should use this option. Remember that the dataset must exist and your job needs permissions to create and remove tables inside that dataset.

      • withFormat

        public BigQueryIO.TypedRead<T> withFormat​(com.google.cloud.bigquery.storage.v1.DataFormat format)
        See DataFormat.

      • withSelectedFields

        public BigQueryIO.TypedRead<T> withSelectedFields​(org.apache.beam.sdk.options.ValueProvider<java.util.List<java.lang.String>> selectedFields)
        Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned.

        Requires BigQueryIO.TypedRead.Method.DIRECT_READ. Not compatible with fromQuery(String).

      • withErrorHandler

        public BigQueryIO.TypedRead<T> withErrorHandler​(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,​?> badRecordErrorHandler)

      • supportsProjectionPushdown

        public boolean supportsProjectionPushdown()
        Specified by:
        supportsProjectionPushdown in interface org.apache.beam.sdk.schemas.ProjectionProducer<T>

      • actuateProjectionPushdown

        public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,​org.apache.beam.sdk.values.PCollection<T>> actuateProjectionPushdown​(java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,​org.apache.beam.sdk.schemas.FieldAccessDescriptor> outputFields)
        Specified by:
        actuateProjectionPushdown in interface org.apache.beam.sdk.schemas.ProjectionProducer<T>