Package org.apache.beam.sdk.io.gcp.bigquery

Class BigQuerySinkMetrics

java.lang.Object

org.apache.beam.sdk.io.gcp.bigquery.BigQuerySinkMetrics

public class BigQuerySinkMetrics
extends java.lang.Object

Helper class to create perworker metrics for BigQuery Sink stages.

In general metrics be in the namespace 'BigQuerySink' and have their name formatted as:

'{baseName}*{metricLabelKey1}:{metricLabelVal1};...{metricLabelKeyN}:{metricLabelValN};'

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	BigQuerySinkMetrics.RpcMethod

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	METRICS_NAMESPACE
static java.lang.String	OK
static java.lang.String	PAYLOAD_TOO_LARGE
static java.lang.String	THROTTLED_TIME
static java.lang.String	UNKNOWN

Constructor Summary

Constructors

Constructor	Description
BigQuerySinkMetrics()

Method Summary

Modifier and Type	Method	Description
static org.apache.beam.sdk.metrics.Counter	appendRowsRowStatusCounter(org.apache.beam.sdk.io.gcp.bigquery.BigQuerySinkMetrics.RowStatus rowStatus, java.lang.String rpcStatus, java.lang.String tableId)
static void	reportFailedRPCMetrics(org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c, BigQuerySinkMetrics.RpcMethod method)	Records RpcRequests counter and RpcLatency histogram for this RPC call.
static void	reportFailedRPCMetrics(org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c, BigQuerySinkMetrics.RpcMethod method, java.lang.String tableId)	Records RpcRequests counter and RpcLatency histogram for this RPC call.
static void	reportSuccessfulRpcMetrics(org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c, BigQuerySinkMetrics.RpcMethod method)	Records RpcRequests counter and RpcLatency histogram for this RPC call.
static void	reportSuccessfulRpcMetrics(org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c, BigQuerySinkMetrics.RpcMethod method, java.lang.String tableId)	Records RpcRequests counter and RpcLatency histogram for this RPC call.
static void	setSupportMetricsDeletion(boolean supportMetricsDeletion)
static void	setSupportStreamingInsertsMetrics(boolean supportStreamingInsertsMetrics)
static org.apache.beam.sdk.metrics.Counter	throttledTimeCounter(BigQuerySinkMetrics.RpcMethod method)
static java.lang.String	throwableToGRPCCodeString(java.lang.Throwable t)	Converts a Throwable to a gRPC Status code.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METRICS_NAMESPACE

public static final java.lang.String METRICS_NAMESPACE

See Also:

Constant Field Values

UNKNOWN

public static final java.lang.String UNKNOWN

OK

public static final java.lang.String OK

PAYLOAD_TOO_LARGE

public static final java.lang.String PAYLOAD_TOO_LARGE

See Also:

Constant Field Values

THROTTLED_TIME

public static final java.lang.String THROTTLED_TIME

See Also:

Constant Field Values

Constructor Detail

BigQuerySinkMetrics

public BigQuerySinkMetrics()

Method Detail

appendRowsRowStatusCounter

public static org.apache.beam.sdk.metrics.Counter appendRowsRowStatusCounter(org.apache.beam.sdk.io.gcp.bigquery.BigQuerySinkMetrics.RowStatus rowStatus,
                                                                             java.lang.String rpcStatus,
                                                                             java.lang.String tableId)

Parameters:

rowStatus - Status of these BigQuery rows.

rpcStatus - rpcStatus

tableId - Table pertaining to the write method. Only included in the metric key if 'supportsMetricsDeletion' is enabled.

Returns:

Metric that tracks the status of BigQuery rows after making an AppendRows RPC call.

throttledTimeCounter

public static org.apache.beam.sdk.metrics.Counter throttledTimeCounter(BigQuerySinkMetrics.RpcMethod method)

Parameters:

method - StorageWriteAPI write method.

Returns:

Counter that tracks throttled time due to RPC retries.

throwableToGRPCCodeString

public static java.lang.String throwableToGRPCCodeString(@Nullable
                                                         java.lang.Throwable t)

Converts a Throwable to a gRPC Status code.

Parameters:

t - Throwable.

Returns:

gRPC status code string or 'UNKNOWN' if 't' is null or does not map to a gRPC error.

reportSuccessfulRpcMetrics

public static void reportSuccessfulRpcMetrics(@Nullable
                                              org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c,
                                              BigQuerySinkMetrics.RpcMethod method)

Records RpcRequests counter and RpcLatency histogram for this RPC call. If 'SupportMetricsDeletion' is enabled, RpcRequests counter will have tableId label set to UNKNOWN. RpcRequets counter will have RpcStatus label set to OK.

Parameters:

c - Context of successful RPC call.

method - StorageWriteAPI method associated with this metric.

reportSuccessfulRpcMetrics

public static void reportSuccessfulRpcMetrics(@Nullable
                                              org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c,
                                              BigQuerySinkMetrics.RpcMethod method,
                                              java.lang.String tableId)

Records RpcRequests counter and RpcLatency histogram for this RPC call. RpcRequets counter will have RpcStatus label set to OK.

Parameters:

c - Context of successful RPC call.

method - StorageWriteAPI method associated with this metric.

tableId - Table pertaining to the write method. Only included in the metric key if 'supportsMetricsDeletion' is enabled.

reportFailedRPCMetrics

public static void reportFailedRPCMetrics(@Nullable
                                          org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c,
                                          BigQuerySinkMetrics.RpcMethod method)

Records RpcRequests counter and RpcLatency histogram for this RPC call. If 'SupportMetricsDeletion' is enabled, RpcRequests counter will have tableId label set to UNKNOWN. RpcRequets counter will have a RpcStatus label set from the gRPC error.

Parameters:

c - Context of successful RPC call.

method - StorageWriteAPI method associated with this metric.

reportFailedRPCMetrics

public static void reportFailedRPCMetrics(@Nullable
                                          org.apache.beam.sdk.io.gcp.bigquery.RetryManager.Operation.Context<?> c,
                                          BigQuerySinkMetrics.RpcMethod method,
                                          java.lang.String tableId)

Records RpcRequests counter and RpcLatency histogram for this RPC call. RpcRequets counter will have a RpcStatus label set from the gRPC error.

Parameters:

c - Context of successful RPC call.

method - StorageWriteAPI method associated with this metric.

tableId - Table pertaining to the write method. Only included in the metric key if 'supportsMetricsDeletion' is enabled.

setSupportStreamingInsertsMetrics

public static void setSupportStreamingInsertsMetrics(boolean supportStreamingInsertsMetrics)

setSupportMetricsDeletion

public static void setSupportMetricsDeletion(boolean supportMetricsDeletion)