Package org.apache.beam.fn.harness.data

Class BeamFnDataGrpcClient

java.lang.Object

org.apache.beam.fn.harness.data.BeamFnDataGrpcClient

All Implemented Interfaces:

BeamFnDataClient

public class BeamFnDataGrpcClient
extends java.lang.Object
implements BeamFnDataClient

A BeamFnDataClient that uses gRPC for sending and receiving data.

TODO: Handle closing clients that are currently not a consumer nor are being consumed.

Constructor Summary

Constructors

Constructor	Description
BeamFnDataGrpcClient(org.apache.beam.sdk.options.PipelineOptions options, java.util.function.Function<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor,org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel> channelFactory, org.apache.beam.sdk.fn.stream.OutboundObserverFactory outboundObserverFactory)

Method Summary

Modifier and Type	Method	Description
org.apache.beam.sdk.fn.data.BeamFnDataOutboundAggregator	createOutboundAggregator(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor apiServiceDescriptor, java.util.function.Supplier<java.lang.String> processBundleRequestIdSupplier, boolean collectElementsIfNoFlushes)	Creates a BeamFnDataOutboundAggregator for buffering and sending outbound data and timers over the data plane.
void	poisonInstructionId(java.lang.String instructionId)	Poisons the instruction id, indicating that future data arriving for it should be discarded.
void	registerReceiver(java.lang.String instructionId, java.util.List<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor> apiServiceDescriptors, org.apache.beam.sdk.fn.data.CloseableFnDataReceiver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> receiver)	Registers a receiver for the provided instruction id.
void	unregisterReceiver(java.lang.String instructionId, java.util.List<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor> apiServiceDescriptors)	Receivers are only expected to be unregistered when bundle processing has completed successfully.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BeamFnDataGrpcClient

public BeamFnDataGrpcClient(org.apache.beam.sdk.options.PipelineOptions options,
                            java.util.function.Function<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor,org.apache.beam.vendor.grpc.v1p69p0.io.grpc.ManagedChannel> channelFactory,
                            org.apache.beam.sdk.fn.stream.OutboundObserverFactory outboundObserverFactory)

Method Detail

registerReceiver

public void registerReceiver(java.lang.String instructionId,
                             java.util.List<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor> apiServiceDescriptors,
                             org.apache.beam.sdk.fn.data.CloseableFnDataReceiver<org.apache.beam.model.fnexecution.v1.BeamFnApi.Elements> receiver)

Description copied from interface: BeamFnDataClient

Registers a receiver for the provided instruction id.

The receiver is not required to be thread safe.

Receivers for successfully processed bundles must be unregistered. See BeamFnDataClient.unregisterReceiver(java.lang.String, java.util.List<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor>) for details.

Any failure during FnDataReceiver.accept(T) will mark the provided instructionId as invalid and will ignore any future data. It is expected that if a bundle fails during processing then the failure will become visible to the BeamFnDataClient during a future FnDataReceiver.accept(T) invocation.

Specified by:

registerReceiver in interface BeamFnDataClient

unregisterReceiver

public void unregisterReceiver(java.lang.String instructionId,
                               java.util.List<org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor> apiServiceDescriptors)

Description copied from interface: BeamFnDataClient

Receivers are only expected to be unregistered when bundle processing has completed successfully.

It is expected that if a bundle fails during processing then the failure will become visible to the BeamFnDataClient during a future FnDataReceiver.accept(T) invocation or via a call to BeamFnDataClient.poisonInstructionId(java.lang.String).

Specified by:

unregisterReceiver in interface BeamFnDataClient

poisonInstructionId

public void poisonInstructionId(java.lang.String instructionId)

Description copied from interface: BeamFnDataClient

Poisons the instruction id, indicating that future data arriving for it should be discarded. Unregisters the receiver if was registered.

Specified by:

poisonInstructionId in interface BeamFnDataClient

createOutboundAggregator

public org.apache.beam.sdk.fn.data.BeamFnDataOutboundAggregator createOutboundAggregator(org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor apiServiceDescriptor,
                                                                                         java.util.function.Supplier<java.lang.String> processBundleRequestIdSupplier,
                                                                                         boolean collectElementsIfNoFlushes)

Description copied from interface: BeamFnDataClient

Creates a BeamFnDataOutboundAggregator for buffering and sending outbound data and timers over the data plane. It is important that BeamFnDataOutboundAggregator.sendOrCollectBufferedDataAndFinishOutboundStreams() is called on the returned BeamFnDataOutboundAggregator at the end of each bundle. If collectElementsIfNoFlushes is set to true, BeamFnDataOutboundAggregator.sendOrCollectBufferedDataAndFinishOutboundStreams() returns the buffered elements instead of sending it through the outbound StreamObserver if there's no previous flush.

Closing the returned aggregator signals the end of the streams.

The returned aggregator is not thread safe.

Specified by:

createOutboundAggregator in interface BeamFnDataClient