Package org.apache.beam.runners.dataflow.options

Interface DataflowPipelineOptions

    • Method Detail

      • getProject

        @Required
@InstanceFactory(org.apache.beam.sdk.extensions.gcp.options.GcpOptions.DefaultProjectFactory.class)
java.lang.String getProject()
        Specified by:
        getProject in interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

      • setProject

        void setProject​(java.lang.String value)
        Specified by:
        setProject in interface org.apache.beam.sdk.extensions.gcp.options.GcpOptions

      • getStagingLocation

        @InstanceFactory(StagingLocationFactory.class)
java.lang.String getStagingLocation()
        GCS path for staging local files, e.g. gs://bucket/object

        Must be a valid Cloud Storage URL, beginning with the prefix "gs://"

        If getStagingLocation() is not set, it will default to GcpOptions.getGcpTempLocation(). GcpOptions.getGcpTempLocation() must be a valid GCS path.

      • setStagingLocation

        void setStagingLocation​(java.lang.String value)

      • isUpdate

        boolean isUpdate()
        Whether to update the currently running pipeline with the same name as this one.

      • setUpdate

        void setUpdate​(boolean value)

      • getCreateFromSnapshot

        java.lang.String getCreateFromSnapshot()
        If set, the snapshot from which the job should be created.

      • setCreateFromSnapshot

        void setCreateFromSnapshot​(java.lang.String value)

      • getTemplateLocation

        java.lang.String getTemplateLocation()
        Where the runner should generate a template file. Must either be local or Cloud Storage.

      • setTemplateLocation

        void setTemplateLocation​(java.lang.String value)
        Sets the Cloud Storage path where the Dataflow template will be stored. Required for creating Flex Templates or Classic Templates.

        Example: 

        
 DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
 options.setTemplateLocation("gs://your-bucket/templates/my-template");
        Parameters:
        value - Cloud Storage path for storing the Dataflow template.

      • getDataflowServiceOptions

        java.util.List<java.lang.String> getDataflowServiceOptions()
        Service options are set by the user and configure the service. This decouples service side feature availability from the Apache Beam release cycle.

      • setDataflowServiceOptions

        void setDataflowServiceOptions​(java.util.List<java.lang.String> options)

      • getServiceAccount

        java.lang.String getServiceAccount()
        Run the job as a specific service account, instead of the default GCE robot.

      • setServiceAccount

        void setServiceAccount​(java.lang.String value)

      • setRegion

        void setRegion​(java.lang.String region)

      • getLabels

        java.util.Map<java.lang.String,​java.lang.String> getLabels()
        Labels that will be applied to the billing records for this job.

      • setLabels

        void setLabels​(java.util.Map<java.lang.String,​java.lang.String> labels)

      • getPipelineUrl

        java.lang.String getPipelineUrl()
        The URL of the staged portable pipeline.

      • setPipelineUrl

        void setPipelineUrl​(java.lang.String urlString)

      • getDataflowWorkerJar

        java.lang.String getDataflowWorkerJar()

      • setDataflowWorkerJar

        void setDataflowWorkerJar​(java.lang.String dataflowWorkerJar)

      • isHotKeyLoggingEnabled

        boolean isHotKeyLoggingEnabled()
        If enabled then the literal key will be logged to Cloud Logging if a hot key is detected.

      • setHotKeyLoggingEnabled

        void setHotKeyLoggingEnabled​(boolean value)

      • getJdkAddOpenModules

        java.util.List<java.lang.String> getJdkAddOpenModules()
        Open modules needed for reflection that access JDK internals with Java 9+

        With JDK 16+, JDK internals are strongly encapsulated and can result in an InaccessibleObjectException being thrown if a tool or library uses reflection that access JDK internals. If you see these errors in your worker logs, you can pass in modules to open using the format module/package=target-module(,target-module)* to allow access to the library. E.g. java.base/java.lang=jamm

        You may see warnings that jamm, a library used to more accurately size objects, is unable to make a private field accessible. To resolve the warning, open the specified module/package to jamm.

      • setJdkAddOpenModules

        void setJdkAddOpenModules​(java.util.List<java.lang.String> options)