Package org.apache.beam.runners.fnexecution.artifact

Class ArtifactStagingService

java.lang.Object

org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService

All Implemented Interfaces:

java.lang.AutoCloseable, org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.AsyncService, org.apache.beam.sdk.fn.server.FnService, org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService

public class ArtifactStagingService
extends org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase
implements org.apache.beam.sdk.fn.server.FnService

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ArtifactStagingService.ArtifactDestination	A pairing of a newly created artifact type and an output stream that will be readable at that type.
static interface	ArtifactStagingService.ArtifactDestinationProvider	Provides a concrete location to which artifacts can be staged on retrieval.

Constructor Summary

Constructors

Constructor	Description
ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Summary

Modifier and Type	Method	Description
static ArtifactStagingService.ArtifactDestinationProvider	beamFilesystemArtifactDestinationProvider(java.lang.String root)	An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.
void	close()
java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>>	getStagedArtifacts(java.lang.String stagingToken)	Returns the rewritten artifacts associated with this job, keyed by environment.
static void	offer(ArtifactRetrievalService retrievalService, org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService, java.lang.String stagingToken)	Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.
void	registerJob(java.lang.String stagingToken, java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts)	Registers a set of artifacts to be staged with this service.
void	removeStagedArtifacts(java.lang.String stagingToken)
org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper>	reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Methods inherited from class org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceImplBase

bindService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.beam.vendor.grpc.v1p69p0.io.grpc.BindableService

bindService

Constructor Detail

ArtifactStagingService

public ArtifactStagingService(ArtifactStagingService.ArtifactDestinationProvider destinationProvider)

Method Detail

registerJob

public void registerJob(java.lang.String stagingToken,
                        java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> artifacts)

Registers a set of artifacts to be staged with this service.

A client (e.g. a Beam SDK) is expected to connect to this service with the given staging token and offer resolution and retrieval of this set of artifacts.

Parameters:

stagingToken - a staging token for this job

artifacts - all artifacts to stage, keyed by environment

getStagedArtifacts

public java.util.Map<java.lang.String,java.util.List<org.apache.beam.model.pipeline.v1.RunnerApi.ArtifactInformation>> getStagedArtifacts(java.lang.String stagingToken)

Returns the rewritten artifacts associated with this job, keyed by environment.

This should be called after the client has finished offering artifacts.

Parameters:

stagingToken - a staging token for this job

removeStagedArtifacts

public void removeStagedArtifacts(java.lang.String stagingToken)
                           throws java.io.IOException

Throws:

java.io.IOException

beamFilesystemArtifactDestinationProvider

public static ArtifactStagingService.ArtifactDestinationProvider beamFilesystemArtifactDestinationProvider(java.lang.String root)

An ArtifactDestinationProvider that places new artifacts as files in a Beam filesystem.

Parameters:

root - the directory in which to place all artifacts

reverseArtifactRetrievalService

public org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactResponseWrapper> reverseArtifactRetrievalService(org.apache.beam.vendor.grpc.v1p69p0.io.grpc.stub.StreamObserver<org.apache.beam.model.jobmanagement.v1.ArtifactApi.ArtifactRequestWrapper> responseObserver)

Specified by:

reverseArtifactRetrievalService in interface org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.AsyncService

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface org.apache.beam.sdk.fn.server.FnService

Throws:

java.lang.Exception

offer

public static void offer(ArtifactRetrievalService retrievalService,
                         org.apache.beam.model.jobmanagement.v1.ArtifactStagingServiceGrpc.ArtifactStagingServiceStub stagingService,
                         java.lang.String stagingToken)
                  throws java.util.concurrent.ExecutionException,
                         java.lang.InterruptedException

Lazily stages artifacts by letting an ArtifactStagingService resolve and request artifacts.

Parameters:

retrievalService - an ArtifactRetrievalService used to resolve and retrieve artifacts

stagingService - an ArtifactStagingService stub which will request artifacts

stagingToken - the staging token of the job whose artifacts will be retrieved

Throws:

java.lang.InterruptedException

java.io.IOException

java.util.concurrent.ExecutionException