Package org.apache.beam.sdk.io.gcp.bigquery

Interface BigQueryOptions

  • All Superinterfaces:
    org.apache.beam.sdk.options.ApplicationNameOptions, org.apache.beam.sdk.extensions.gcp.options.GcpOptions, org.apache.beam.sdk.extensions.gcp.options.GoogleApiDebugOptions, org.apache.beam.sdk.transforms.display.HasDisplayData, org.apache.beam.sdk.options.PipelineOptions, org.apache.beam.sdk.options.StreamingOptions
    All Known Subinterfaces:
    TestBigQueryOptions
    public interface BigQueryOptions
extends org.apache.beam.sdk.options.ApplicationNameOptions, org.apache.beam.sdk.extensions.gcp.options.GcpOptions, org.apache.beam.sdk.options.PipelineOptions, org.apache.beam.sdk.options.StreamingOptions
    Properties needed when using Google BigQuery with the Apache Beam SDK.

    • Method Detail

      • getTempDatasetId

        @String("bigquery.googleapis.com/cloud_dataflow")
java.lang.String getTempDatasetId()

      • setTempDatasetId

        void setTempDatasetId​(java.lang.String value)

      • getHTTPReadTimeout

        @Integer(80000)
java.lang.Integer getHTTPReadTimeout()

      • setHTTPReadTimeout

        void setHTTPReadTimeout​(java.lang.Integer timeout)

      • getHTTPWriteTimeout

        @Integer(900000)
java.lang.Integer getHTTPWriteTimeout()

      • setHTTPWriteTimeout

        void setHTTPWriteTimeout​(java.lang.Integer timeout)

      • getInsertBundleParallelism

        @Integer(3)
java.lang.Integer getInsertBundleParallelism()

      • setInsertBundleParallelism

        void setInsertBundleParallelism​(java.lang.Integer parallelism)

      • getNumStreamingKeys

        @Integer(50)
java.lang.Integer getNumStreamingKeys()

      • setNumStreamingKeys

        void setNumStreamingKeys​(java.lang.Integer value)

      • getMaxStreamingRowsToBatch

        @Long(500L)
java.lang.Long getMaxStreamingRowsToBatch()

      • setMaxStreamingRowsToBatch

        void setMaxStreamingRowsToBatch​(java.lang.Long value)

      • getMaxStreamingBatchSize

        @Long(65536L)
java.lang.Long getMaxStreamingBatchSize()

      • setMaxStreamingBatchSize

        void setMaxStreamingBatchSize​(java.lang.Long value)

      • getBqStreamingApiLoggingFrequencySec

        @Integer(180)
java.lang.Integer getBqStreamingApiLoggingFrequencySec()

      • setBqStreamingApiLoggingFrequencySec

        void setBqStreamingApiLoggingFrequencySec​(java.lang.Integer value)

      • getUseStorageWriteApi

        @Boolean(false)
java.lang.Boolean getUseStorageWriteApi()

      • setUseStorageWriteApi

        void setUseStorageWriteApi​(java.lang.Boolean value)

      • getUseStorageWriteApiAtLeastOnce

        @Boolean(false)
java.lang.Boolean getUseStorageWriteApiAtLeastOnce()

      • setUseStorageWriteApiAtLeastOnce

        void setUseStorageWriteApiAtLeastOnce​(java.lang.Boolean value)

      • getNumStorageWriteApiStreams

        @Integer(0)
java.lang.Integer getNumStorageWriteApiStreams()

      • setNumStorageWriteApiStreams

        void setNumStorageWriteApiStreams​(java.lang.Integer value)

      • getNumStorageWriteApiStreamAppendClients

        @Integer(1)
java.lang.Integer getNumStorageWriteApiStreamAppendClients()

      • setNumStorageWriteApiStreamAppendClients

        void setNumStorageWriteApiStreamAppendClients​(java.lang.Integer value)

      • getMinConnectionPoolConnections

        @Integer(2)
java.lang.Integer getMinConnectionPoolConnections()

      • setMinConnectionPoolConnections

        void setMinConnectionPoolConnections​(java.lang.Integer value)

      • getMaxConnectionPoolConnections

        @Integer(20)
java.lang.Integer getMaxConnectionPoolConnections()

      • setMaxConnectionPoolConnections

        void setMaxConnectionPoolConnections​(java.lang.Integer value)

      • getStorageWriteMaxInflightRequests

        @Long(1000L)
java.lang.Long getStorageWriteMaxInflightRequests()

      • setStorageWriteMaxInflightRequests

        void setStorageWriteMaxInflightRequests​(java.lang.Long value)

      • getStorageWriteMaxInflightBytes

        @Long(104857600L)
java.lang.Long getStorageWriteMaxInflightBytes()

      • setStorageWriteMaxInflightBytes

        void setStorageWriteMaxInflightBytes​(java.lang.Long value)

      • getUseStorageApiConnectionPool

        @Boolean(false)
java.lang.Boolean getUseStorageApiConnectionPool()

      • setUseStorageApiConnectionPool

        void setUseStorageApiConnectionPool​(java.lang.Boolean value)

      • getStorageWriteApiTriggeringFrequencySec

        java.lang.Integer getStorageWriteApiTriggeringFrequencySec()

      • setStorageWriteApiTriggeringFrequencySec

        void setStorageWriteApiTriggeringFrequencySec​(java.lang.Integer value)

      • getStorageWriteApiMaxRetries

        @Integer(500)
java.lang.Integer getStorageWriteApiMaxRetries()

      • setStorageWriteApiMaxRetries

        void setStorageWriteApiMaxRetries​(java.lang.Integer value)

      • getMaxBufferingDurationMilliSec

        @Integer(0)
java.lang.Integer getMaxBufferingDurationMilliSec()

      • setMaxBufferingDurationMilliSec

        void setMaxBufferingDurationMilliSec​(java.lang.Integer value)

      • getBigQueryProject

        java.lang.String getBigQueryProject()

      • setBigQueryProject

        void setBigQueryProject​(java.lang.String value)

      • getStorageApiAppendThresholdBytes

        @Integer(2097152)
java.lang.Integer getStorageApiAppendThresholdBytes()

      • setStorageApiAppendThresholdBytes

        void setStorageApiAppendThresholdBytes​(java.lang.Integer value)

      • getStorageApiAppendThresholdRecordCount

        @Integer(150000)
java.lang.Integer getStorageApiAppendThresholdRecordCount()

      • setStorageApiAppendThresholdRecordCount

        void setStorageApiAppendThresholdRecordCount​(java.lang.Integer value)

      • getStorageWriteApiMaxRequestSize

        @Long(10000000L)
java.lang.Long getStorageWriteApiMaxRequestSize()

      • setStorageWriteApiMaxRequestSize

        void setStorageWriteApiMaxRequestSize​(java.lang.Long value)

      • getEnableStorageReadApiV2

        @Boolean(false)
java.lang.Boolean getEnableStorageReadApiV2()

      • setEnableStorageReadApiV2

        void setEnableStorageReadApiV2​(java.lang.Boolean value)

      • getJobLabelsMap

        java.util.Map<java.lang.String,​java.lang.String> getJobLabelsMap()

      • setJobLabelsMap

        void setJobLabelsMap​(java.util.Map<java.lang.String,​java.lang.String> value)

      • getBigQueryEndpoint

        @Hidden
java.lang.String getBigQueryEndpoint()
        BQ endpoint to use. If unspecified, uses the default endpoint.

      • setBigQueryEndpoint

        void setBigQueryEndpoint​(java.lang.String value)

      • getGroupFilesFileLoad

        @Hidden
@Boolean(false)
java.lang.Boolean getGroupFilesFileLoad()
        Choose to use a GBK when gathering a list of files in batch FILE_LOAD.

        The purpose of this option is to accommodate the runner compatibility, for example, some runners having known issues on large side input, turning on this option avoids trigger side input related issues.

        This is an experimental pipeline option, no backward compatibility guaranteed.

      • setGroupFilesFileLoad

        void setGroupFilesFileLoad​(java.lang.Boolean value)