com.google.api.services.dataflow.model

Class WorkerPool

java.lang.Object

java.util.AbstractMap<String,Object>

com.google.api.client.util.GenericData

com.google.api.client.json.GenericJson

com.google.api.services.dataflow.model.WorkerPool

All Implemented Interfaces:

Cloneable, Map<String,Object>

public final class WorkerPool
extends GenericJson

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

This is the Java data model class that specifies how to parse/serialize into the JSON that is transmitted over HTTP when working with the Dataflow API. For a detailed explanation see: https://developers.google.com/api-client-library/java/google-http-java-client/json

Author:

Google, Inc.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.api.client.util.GenericData

GenericData.Flags

Nested classes/interfaces inherited from class java.util.AbstractMap

AbstractMap.SimpleEntry<K,V>, AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface java.util.Map

Map.Entry<K,V>

Constructor Summary

Constructors

Constructor and Description
WorkerPool()

Method Summary

Modifier and Type	Method and Description
WorkerPool	clone()
AutoscalingSettings	getAutoscalingSettings() Settings for autoscaling of this WorkerPool.
List<Disk>	getDataDisks() Data disks that are used by a VM in this workflow.
String	getDefaultPackageSet() The default package set to install.
Integer	getDiskSizeGb() Size of root disk for VMs, in GB.
String	getDiskSourceImage() Fully qualified source image for disks.
String	getDiskType() Type of root disk for VMs.
String	getIpConfiguration() Configuration for VM IPs.
String	getKind() The kind of the worker pool; currently only `harness` and `shuffle` are supported.
String	getMachineType() Machine type (e.g.
Map<String,String>	getMetadata() Metadata to set on the Google Compute Engine VMs.
String	getNetwork() Network to which VMs will be assigned.
Integer	getNumThreadsPerWorker() The number of threads per worker harness.
Integer	getNumWorkers() Number of Google Compute Engine workers in this pool needed to execute the job.
String	getOnHostMaintenance() The action to take on host maintenance, as defined by the Google Compute Engine API.
List<DataflowPackage>	getPackages() Packages to be installed on workers.
Map<String,Object>	getPoolArgs() Extra arguments for this worker pool.
List<SdkHarnessContainerImage>	getSdkHarnessContainerImages() Set of SDK harness containers needed to execute this pipeline.
String	getSubnetwork() Subnetwork to which VMs will be assigned, if desired.
TaskRunnerSettings	getTaskrunnerSettings() Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner.
String	getTeardownPolicy() Sets the policy for determining when to turndown worker pool.
String	getWorkerHarnessContainerImage() Required.
String	getZone() Zone to run the worker pools in.
WorkerPool	set(String fieldName, Object value)
WorkerPool	setAutoscalingSettings(AutoscalingSettings autoscalingSettings) Settings for autoscaling of this WorkerPool.
WorkerPool	setDataDisks(List<Disk> dataDisks) Data disks that are used by a VM in this workflow.
WorkerPool	setDefaultPackageSet(String defaultPackageSet) The default package set to install.
WorkerPool	setDiskSizeGb(Integer diskSizeGb) Size of root disk for VMs, in GB.
WorkerPool	setDiskSourceImage(String diskSourceImage) Fully qualified source image for disks.
WorkerPool	setDiskType(String diskType) Type of root disk for VMs.
WorkerPool	setIpConfiguration(String ipConfiguration) Configuration for VM IPs.
WorkerPool	setKind(String kind) The kind of the worker pool; currently only `harness` and `shuffle` are supported.
WorkerPool	setMachineType(String machineType) Machine type (e.g.
WorkerPool	setMetadata(Map<String,String> metadata) Metadata to set on the Google Compute Engine VMs.
WorkerPool	setNetwork(String network) Network to which VMs will be assigned.
WorkerPool	setNumThreadsPerWorker(Integer numThreadsPerWorker) The number of threads per worker harness.
WorkerPool	setNumWorkers(Integer numWorkers) Number of Google Compute Engine workers in this pool needed to execute the job.
WorkerPool	setOnHostMaintenance(String onHostMaintenance) The action to take on host maintenance, as defined by the Google Compute Engine API.
WorkerPool	setPackages(List<DataflowPackage> packages) Packages to be installed on workers.
WorkerPool	setPoolArgs(Map<String,Object> poolArgs) Extra arguments for this worker pool.
WorkerPool	setSdkHarnessContainerImages(List<SdkHarnessContainerImage> sdkHarnessContainerImages) Set of SDK harness containers needed to execute this pipeline.
WorkerPool	setSubnetwork(String subnetwork) Subnetwork to which VMs will be assigned, if desired.
WorkerPool	setTaskrunnerSettings(TaskRunnerSettings taskrunnerSettings) Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner.
WorkerPool	setTeardownPolicy(String teardownPolicy) Sets the policy for determining when to turndown worker pool.
WorkerPool	setWorkerHarnessContainerImage(String workerHarnessContainerImage) Required.
WorkerPool	setZone(String zone) Zone to run the worker pools in.

Methods inherited from class com.google.api.client.json.GenericJson

getFactory, setFactory, toPrettyString, toString

Methods inherited from class com.google.api.client.util.GenericData

entrySet, equals, get, getClassInfo, getUnknownKeys, hashCode, put, putAll, remove, setUnknownKeys

Methods inherited from class java.util.AbstractMap

clear, containsKey, containsValue, isEmpty, keySet, size, values

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

compute, computeIfAbsent, computeIfPresent, forEach, getOrDefault, merge, putIfAbsent, remove, replace, replace, replaceAll

Constructor Detail

WorkerPool

public WorkerPool()

Method Detail

getAutoscalingSettings

public AutoscalingSettings getAutoscalingSettings()

Settings for autoscaling of this WorkerPool.

Returns:

value or null for none

setAutoscalingSettings

public WorkerPool setAutoscalingSettings(AutoscalingSettings autoscalingSettings)

Settings for autoscaling of this WorkerPool.

Parameters:

autoscalingSettings - autoscalingSettings or null for none

getDataDisks

public List<Disk> getDataDisks()

Data disks that are used by a VM in this workflow.

Returns:

value or null for none

setDataDisks

public WorkerPool setDataDisks(List<Disk> dataDisks)

Data disks that are used by a VM in this workflow.

Parameters:

dataDisks - dataDisks or null for none

getDefaultPackageSet

public String getDefaultPackageSet()

The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language.

Returns:

value or null for none

setDefaultPackageSet

public WorkerPool setDefaultPackageSet(String defaultPackageSet)

The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language.

Parameters:

defaultPackageSet - defaultPackageSet or null for none

getDiskSizeGb

public Integer getDiskSizeGb()

Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setDiskSizeGb

public WorkerPool setDiskSizeGb(Integer diskSizeGb)

Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default.

Parameters:

diskSizeGb - diskSizeGb or null for none

getDiskSourceImage

public String getDiskSourceImage()

Fully qualified source image for disks.

Returns:

value or null for none

setDiskSourceImage

public WorkerPool setDiskSourceImage(String diskSourceImage)

Fully qualified source image for disks.

Parameters:

diskSourceImage - diskSourceImage or null for none

getDiskType

public String getDiskType()

Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setDiskType

public WorkerPool setDiskType(String diskType)

Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default.

Parameters:

diskType - diskType or null for none

getIpConfiguration

public String getIpConfiguration()

Configuration for VM IPs.

Returns:

value or null for none

setIpConfiguration

public WorkerPool setIpConfiguration(String ipConfiguration)

Configuration for VM IPs.

Parameters:

ipConfiguration - ipConfiguration or null for none

getKind

public String getKind()

The kind of the worker pool; currently only `harness` and `shuffle` are supported.

Returns:

value or null for none

setKind

public WorkerPool setKind(String kind)

The kind of the worker pool; currently only `harness` and `shuffle` are supported.

Parameters:

kind - kind or null for none

getMachineType

public String getMachineType()

Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setMachineType

public WorkerPool setMachineType(String machineType)

Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default.

Parameters:

machineType - machineType or null for none

getMetadata

public Map<String,String> getMetadata()

Metadata to set on the Google Compute Engine VMs.

Returns:

value or null for none

setMetadata

public WorkerPool setMetadata(Map<String,String> metadata)

Metadata to set on the Google Compute Engine VMs.

Parameters:

metadata - metadata or null for none

getNetwork

public String getNetwork()

Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default".

Returns:

value or null for none

setNetwork

public WorkerPool setNetwork(String network)

Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default".

Parameters:

network - network or null for none

getNumThreadsPerWorker

public Integer getNumThreadsPerWorker()

The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming).

Returns:

value or null for none

setNumThreadsPerWorker

public WorkerPool setNumThreadsPerWorker(Integer numThreadsPerWorker)

The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming).

Parameters:

numThreadsPerWorker - numThreadsPerWorker or null for none

getNumWorkers

public Integer getNumWorkers()

Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setNumWorkers

public WorkerPool setNumWorkers(Integer numWorkers)

Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default.

Parameters:

numWorkers - numWorkers or null for none

getOnHostMaintenance

public String getOnHostMaintenance()

The action to take on host maintenance, as defined by the Google Compute Engine API.

Returns:

value or null for none

setOnHostMaintenance

public WorkerPool setOnHostMaintenance(String onHostMaintenance)

The action to take on host maintenance, as defined by the Google Compute Engine API.

Parameters:

onHostMaintenance - onHostMaintenance or null for none

getPackages

public List<DataflowPackage> getPackages()

Packages to be installed on workers.

Returns:

value or null for none

setPackages

public WorkerPool setPackages(List<DataflowPackage> packages)

Packages to be installed on workers.

Parameters:

packages - packages or null for none

getPoolArgs

public Map<String,Object> getPoolArgs()

Extra arguments for this worker pool.

Returns:

value or null for none

setPoolArgs

public WorkerPool setPoolArgs(Map<String,Object> poolArgs)

Extra arguments for this worker pool.

Parameters:

poolArgs - poolArgs or null for none

getSdkHarnessContainerImages

public List<SdkHarnessContainerImage> getSdkHarnessContainerImages()

Set of SDK harness containers needed to execute this pipeline. This will only be set in the Fn API path. For non-cross-language pipelines this should have only one entry. Cross-language pipelines will have two or more entries.

Returns:

value or null for none

setSdkHarnessContainerImages

public WorkerPool setSdkHarnessContainerImages(List<SdkHarnessContainerImage> sdkHarnessContainerImages)

Set of SDK harness containers needed to execute this pipeline. This will only be set in the Fn API path. For non-cross-language pipelines this should have only one entry. Cross-language pipelines will have two or more entries.

Parameters:

sdkHarnessContainerImages - sdkHarnessContainerImages or null for none

getSubnetwork

public String getSubnetwork()

Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK".

Returns:

value or null for none

setSubnetwork

public WorkerPool setSubnetwork(String subnetwork)

Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK".

Parameters:

subnetwork - subnetwork or null for none

getTaskrunnerSettings

public TaskRunnerSettings getTaskrunnerSettings()

Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner. Users should ignore this field.

Returns:

value or null for none

setTaskrunnerSettings

public WorkerPool setTaskrunnerSettings(TaskRunnerSettings taskrunnerSettings)

Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner. Users should ignore this field.

Parameters:

taskrunnerSettings - taskrunnerSettings or null for none

getTeardownPolicy

public String getTeardownPolicy()

Sets the policy for determining when to turndown worker pool. Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and `TEARDOWN_NEVER`. `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn down. If the workers are not torn down by the service, they will continue to run and use Google Compute Engine VM resources in the user's project until they are explicitly terminated by the user. Because of this, Google recommends using the `TEARDOWN_ALWAYS` policy except for small, manually supervised test jobs. If unknown or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setTeardownPolicy

public WorkerPool setTeardownPolicy(String teardownPolicy)

Sets the policy for determining when to turndown worker pool. Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and `TEARDOWN_NEVER`. `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn down. If the workers are not torn down by the service, they will continue to run and use Google Compute Engine VM resources in the user's project until they are explicitly terminated by the user. Because of this, Google recommends using the `TEARDOWN_ALWAYS` policy except for small, manually supervised test jobs. If unknown or unspecified, the service will attempt to choose a reasonable default.

Parameters:

teardownPolicy - teardownPolicy or null for none

getWorkerHarnessContainerImage

public String getWorkerHarnessContainerImage()

Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry. Deprecated for the Fn API path. Use sdk_harness_container_images instead.

Returns:

value or null for none

setWorkerHarnessContainerImage

public WorkerPool setWorkerHarnessContainerImage(String workerHarnessContainerImage)

Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry. Deprecated for the Fn API path. Use sdk_harness_container_images instead.

Parameters:

workerHarnessContainerImage - workerHarnessContainerImage or null for none

getZone

public String getZone()

Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default.

Returns:

value or null for none

setZone

public WorkerPool setZone(String zone)

Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default.

Parameters:

zone - zone or null for none

set

public WorkerPool set(String fieldName,
                      Object value)

Overrides:

set in class GenericJson

clone

public WorkerPool clone()

Overrides:

clone in class GenericJson