com.microsoft.azure.kusto.ingest

Class ManagedStreamingIngestClient

java.lang.Object

com.microsoft.azure.kusto.ingest.IngestClientBase

com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient

All Implemented Interfaces:

IngestClient, QueuedIngestClient, Closeable, AutoCloseable

public class ManagedStreamingIngestClient
extends IngestClientBase
implements QueuedIngestClient

ManagedStreamingIngestClient

This class combines a managed streaming client with a queued streaming client, to create an optimized experience. Since the streaming client communicates directly with the engine, it's more prone to failure, so this class holds both a streaming client and a queued client. It retries 2 times using the streaming client, after which it falls back to the queued streaming client in case of failure. By default the policy for choosing a queued ingestion on the first try is the checking of weather the size of the estimated raw stream size (a conversion to compressed CSV) is bigger than 4MB, it will fall back to the queued streaming client. Use setQueuingPolicyFactor(double) to override the predicate heuristics. Use SourceInfo.setRawSizeInBytes to set the raw size of the data.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CLASS_NAME
static int	RETRY_COUNT

Constructor Summary

Constructors

Constructor and Description
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder) Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder) instead.
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, @Nullable HttpClientProperties properties)
ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder, ConnectionStringBuilder queryEndpointConnectionStringBuilder, @Nullable HttpClientProperties properties, boolean autoCorrectEndpoint) Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder, HttpClientProperties) instead. This constructor should only be used for advanced cases. If your endpoints are standard, or you do not know, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties))} instead.
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable com.azure.core.http.HttpClient httpClient)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable com.azure.core.http.HttpClient httpClient, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable HttpClientProperties properties)
ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder, @Nullable HttpClientProperties properties, boolean autoCorrectEndpoint)
ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager, AzureStorageClient storageClient, StreamingClient streamingClient) Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder) instead.
ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager, AzureStorageClient storageClient, StreamingClient streamingClient, ExponentialRetry retryTemplate)

Method Summary

Modifier and Type	Method and Description
void	close()
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString) Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string, with default http client properties. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)
static ManagedStreamingIngestClient	fromDmConnectionString(ConnectionStringBuilder dmConnectionString, @Nullable HttpClientProperties properties) Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString) Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string, with default http client properties. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)
static ManagedStreamingIngestClient	fromEngineConnectionString(ConnectionStringBuilder engineConnectionString, @Nullable HttpClientProperties properties) Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)
protected String	getClientType()
IngestionResourceManager	getResourceManager()
protected reactor.core.publisher.Mono<IngestionResult>	ingestFromBlobAsyncImpl(BlobSourceInfo blobSourceInfo, IngestionProperties ingestionProperties)
protected reactor.core.publisher.Mono<IngestionResult>	ingestFromFileAsyncImpl(FileSourceInfo fileSourceInfo, IngestionProperties ingestionProperties)
protected reactor.core.publisher.Mono<IngestionResult>	ingestFromResultSetAsyncImpl(ResultSetSourceInfo resultSetSourceInfo, IngestionProperties ingestionProperties)
protected reactor.core.publisher.Mono<IngestionResult>	ingestFromStreamAsyncImpl(StreamSourceInfo streamSourceInfo, IngestionProperties ingestionProperties)
void	setQueueRequestOptions(com.azure.storage.common.policy.RequestRetryOptions queueRequestOptions) Setter for QueueRequestOptions used by the client on adding ingest message to the Azure queue, read here https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#ingest-client-flavors about Kusto queued ingestion
void	setQueuingPolicyFactor(double factor) setQueuingPolicyFactor

Methods inherited from class com.microsoft.azure.kusto.ingest.IngestClientBase

getIngestionTraceAttributes, ingestFromBlob, ingestFromBlobAsync, ingestFromFile, ingestFromFileAsync, ingestFromResultSet, ingestFromResultSetAsync, ingestFromStream, ingestFromStreamAsync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.microsoft.azure.kusto.ingest.IngestClient

ingestFromBlob, ingestFromBlobAsync, ingestFromFile, ingestFromFileAsync, ingestFromResultSet, ingestFromResultSetAsync, ingestFromStream, ingestFromStreamAsync

Field Detail

RETRY_COUNT

public static final int RETRY_COUNT

See Also:

Constant Field Values

CLASS_NAME

public static final String CLASS_NAME

Constructor Detail

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder)
                             throws URISyntaxException

Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder) instead.

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder, HttpClientProperties) instead. This constructor should only be used for advanced cases. If your endpoints are standard, or you do not know, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties))} instead.

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

properties - - Additional properties to configure the http client

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable com.azure.core.http.HttpClient httpClient,
                                    boolean autoCorrectEndpoint)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

httpClient - - HTTP client

autoCorrectEndpoint - - Flag to indicate whether to correct the endpoint URI or not

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder ingestionEndpointConnectionStringBuilder,
                                    ConnectionStringBuilder queryEndpointConnectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties)
                             throws URISyntaxException

Parameters:

ingestionEndpointConnectionStringBuilder - - Endpoint for ingesting data, usually starts with "https://ingest-"

queryEndpointConnectionStringBuilder - - Endpoint for querying data, does not include "ingest-"

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable HttpClientProperties properties)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

properties - - Additional properties to configure the http client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(ConnectionStringBuilder connectionStringBuilder,
                                    @Nullable
                                    @Nullable com.azure.core.http.HttpClient httpClient)
                             throws URISyntaxException

Parameters:

connectionStringBuilder - - Client connection string

httpClient - - HTTP client

Throws:

URISyntaxException - if the connection string is invalid

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager,
                                    AzureStorageClient storageClient,
                                    StreamingClient streamingClient)

Deprecated. - This method is slated to be private. Use IngestClientFactory.createManagedStreamingIngestClient(ConnectionStringBuilder) instead.

Parameters:

resourceManager - ingestion resources manager

storageClient - - storage utilities

streamingClient - - the streaming client

ManagedStreamingIngestClient

public ManagedStreamingIngestClient(com.microsoft.azure.kusto.ingest.ResourceManager resourceManager,
                                    AzureStorageClient storageClient,
                                    StreamingClient streamingClient,
                                    ExponentialRetry retryTemplate)

Parameters:

resourceManager - ingestion resources manager

storageClient - - storage utilities

streamingClient - - the streaming client

retryTemplate - - retry template

Method Detail

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString)
                                                           throws URISyntaxException

Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string, with default http client properties. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromDmConnectionString

public static ManagedStreamingIngestClient fromDmConnectionString(ConnectionStringBuilder dmConnectionString,
                                                                  @Nullable
                                                                  @Nullable HttpClientProperties properties)
                                                           throws URISyntaxException

Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from a DM connection string. This method infers the engine connection string from the DM connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

dmConnectionString - dm connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString)
                                                               throws URISyntaxException

Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string, with default http client properties. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

fromEngineConnectionString

public static ManagedStreamingIngestClient fromEngineConnectionString(ConnectionStringBuilder engineConnectionString,
                                                                      @Nullable
                                                                      @Nullable HttpClientProperties properties)
                                                               throws URISyntaxException

Deprecated. - Ingest clients now automatically deduce the endpoint, use ManagedStreamingIngestClient(ConnectionStringBuilder, HttpClientProperties) instead. Creates a new ManagedStreamingIngestClient from an engine connection string. This method infers the DM connection string from the engine connection string. For advanced usage, use ManagedStreamingIngestClient(ConnectionStringBuilder, ConnectionStringBuilder)

Parameters:

engineConnectionString - engine connection string

properties - additional properties to configure the http client

Returns:

a new ManagedStreamingIngestClient

Throws:

URISyntaxException - if the connection string is invalid

ingestFromFileAsyncImpl

protected reactor.core.publisher.Mono<IngestionResult> ingestFromFileAsyncImpl(FileSourceInfo fileSourceInfo,
                                                                               IngestionProperties ingestionProperties)

Specified by:

ingestFromFileAsyncImpl in class IngestClientBase

ingestFromBlobAsyncImpl

protected reactor.core.publisher.Mono<IngestionResult> ingestFromBlobAsyncImpl(BlobSourceInfo blobSourceInfo,
                                                                               IngestionProperties ingestionProperties)

Specified by:

ingestFromBlobAsyncImpl in class IngestClientBase

ingestFromResultSetAsyncImpl

protected reactor.core.publisher.Mono<IngestionResult> ingestFromResultSetAsyncImpl(ResultSetSourceInfo resultSetSourceInfo,
                                                                                    IngestionProperties ingestionProperties)

Specified by:

ingestFromResultSetAsyncImpl in class IngestClientBase

ingestFromStreamAsyncImpl

protected reactor.core.publisher.Mono<IngestionResult> ingestFromStreamAsyncImpl(StreamSourceInfo streamSourceInfo,
                                                                                 IngestionProperties ingestionProperties)

Specified by:

ingestFromStreamAsyncImpl in class IngestClientBase

setQueuingPolicyFactor

public void setQueuingPolicyFactor(double factor)

setQueuingPolicyFactor

A factor used to tune the policy that handles the logic over which data size would the client choose to directly use queued ingestion instead of trying streaming ingestion first. Setting the factor will create a new ManagedStreamingQueuingPolicy with this factor, which will be used in the future ingestion calls.

Parameters:

factor - - Default is 1.

getClientType

protected String getClientType()

Specified by:

getClientType in class IngestClientBase

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

setQueueRequestOptions

public void setQueueRequestOptions(com.azure.storage.common.policy.RequestRetryOptions queueRequestOptions)

Description copied from interface: QueuedIngestClient

Setter for QueueRequestOptions used by the client on adding ingest message to the Azure queue, read here https://docs.microsoft.com/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#ingest-client-flavors about Kusto queued ingestion

Specified by:

setQueueRequestOptions in interface QueuedIngestClient

Parameters:

queueRequestOptions - - Options to use when creating QueueClient

getResourceManager

public IngestionResourceManager getResourceManager()

Specified by:

getResourceManager in interface QueuedIngestClient