Package org.apache.beam.runners.dataflow

Provides a Beam runner that executes pipelines on the Google Cloud Dataflow service.

Interface Summary

Interface	Description
TestDataflowPipelineOptions	A set of options used to configure the TestPipeline.
TransformTranslator<TransformT extends org.apache.beam.sdk.transforms.PTransform>	A TransformTranslator knows how to translate a particular subclass of PTransform for the Cloud Dataflow service.
TransformTranslator.StepTranslationContext	The interface for a TransformTranslator to build a Dataflow step.
TransformTranslator.TranslationContext	The interface provided to registered callbacks for interacting with the DataflowRunner, including reading and writing the values of PCollections and side inputs.

Class Summary

Class	Description
BatchStatefulParDoOverrides	PTransformOverrideFactories that expands to correctly implement stateful ParDo using window-unaware BatchViewOverrides.GroupByKeyAndSortValuesOnly to linearize processing per key.
BatchStatefulParDoOverrides.BatchStatefulDoFn<K,V,OutputT>	A key-preserving DoFn that explodes an iterable that has been grouped by key and window.
CreateDataflowView<ElemT,ViewT>	A DataflowRunner marker class for creating a PCollectionView.
DataflowClient	Wrapper around the generated Dataflow client to provide common functionality.
DataflowPipelineJob	A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.
DataflowPipelineRegistrar	Contains the PipelineOptionsRegistrar and PipelineRunnerRegistrar for the DataflowRunner.
DataflowPipelineRegistrar.Options	Register the DataflowPipelineOptions.
DataflowPipelineRegistrar.Runner	Register the DataflowRunner.
DataflowPipelineTranslator	DataflowPipelineTranslator knows how to translate Pipeline objects into Cloud Dataflow Service API Jobs.
DataflowPipelineTranslator.JobSpecification	The result of a job translation.
DataflowRunner	A PipelineRunner that executes the operations in the pipeline by first translating them to the Dataflow representation using the DataflowPipelineTranslator and then submitting them to a Dataflow service for execution.
DataflowRunner.DataflowTransformTranslator
DataflowRunner.StreamingPCollectionViewWriterFn<T>	A marker DoFn for writing the contents of a PCollection to a streaming PCollectionView backend implementation.
DataflowRunnerHooks	An instance of this class can be passed to the DataflowRunner to add user defined hooks to be invoked at various times during pipeline execution.
DataflowRunnerInfo	Populates versioning and other information for DataflowRunner.
GroupIntoBatchesOverride
PrimitiveParDoSingleFactory<InputT,OutputT>	A PTransformOverrideFactory that produces PrimitiveParDoSingleFactory.ParDoSingle instances from ParDo.SingleOutput instances.
PrimitiveParDoSingleFactory.ParDoSingle<InputT,OutputT>	A single-output primitive ParDo.
PrimitiveParDoSingleFactory.PayloadTranslator	A translator for PrimitiveParDoSingleFactory.ParDoSingle.
PrimitiveParDoSingleFactory.Registrar	Registers PrimitiveParDoSingleFactory.PayloadTranslator.
TestDataflowRunner	TestDataflowRunner is a pipeline runner that wraps a DataflowRunner when running tests against the TestPipeline.

Exception Summary

Exception	Description
DataflowJobAlreadyExistsException	An exception that is thrown if the unique job name constraint of the Dataflow service is broken because an existing job with the same job name is currently active.
DataflowJobAlreadyUpdatedException	An exception that is thrown if the existing job has already been updated within the Dataflow service and is no longer able to be updated.
DataflowJobException	A RuntimeException that contains information about a DataflowPipelineJob.
DataflowServiceException	Signals there was an error retrieving information about a job from the Cloud Dataflow Service.