net.snowflake.client.jdbc.cloud.storage

Interface SnowflakeStorageClient

All Known Implementing Classes:

SnowflakeAzureClient, SnowflakeGCSClient, SnowflakeS3Client

public interface SnowflakeStorageClient

Interface for storage client provider implementations

Author:

lgiakoumakis

Method Summary

Modifier and Type	Method and Description
void	addDigestMetadata(StorageObjectMetadata meta, String digest) Adds digest metadata to the StorageObjectMetadata object
void	addEncryptionMetadata(StorageObjectMetadata meta, MatDesc matDesc, byte[] ivData, byte[] encKeK, long contentLength) Adds encryption metadata to the StorageObjectMetadata object
void	addStreamingIngestMetadata(StorageObjectMetadata meta, String clientName, String clientKey) Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation
void	download(SFSession connection, String command, String localLocation, String destFileName, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from remote storage.
InputStream	downloadToStream(SFSession connection, String command, int parallelism, String remoteStorageLocation, String stageFilePath, String stageRegion, String presignedUrl) Download a file from remote storage
String	getDigestMetadata(StorageObjectMetadata meta) Gets digest metadata to the StorageObjectMetadata object
int	getEncryptionKeySize()
String	getMatdescKey() Returns the material descriptor key
int	getMaxRetries()
StorageObjectMetadata	getObjectMetadata(String remoteStorageLocation, String prefix) Returns the metadata properties for a remote storage object
int	getRetryBackoffMaxExponent() Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry
int	getRetryBackoffMin()
String	getStreamingIngestClientKey(StorageObjectMetadata meta) Gets streaming ingest client key to the StorageObjectMetadata object
String	getStreamingIngestClientName(StorageObjectMetadata meta) Gets streaming ingest client name to the StorageObjectMetadata object
void	handleStorageException(Exception ex, int retryCount, String operation, SFSession connection, String command) Handles exceptions thrown by the remote storage provider
boolean	isEncrypting()
StorageObjectSummaryCollection	listObjects(String remoteStorageLocation, String prefix) For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects
void	renew(Map<?,?> stageCredentials) Re-creates the encapsulated storage client with a fresh access token
default boolean	requirePresignedUrl()
void	shutdown() shuts down the client
void	upload(SFSession connection, String command, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl) Upload a file (-stream) to remote storage
default void	uploadWithPresignedUrlWithoutConnection(int networkTimeoutInMilli, HttpClientSettingsKey ocspModeAndProxyKey, int parallelism, boolean uploadFromStream, String remoteStorageLocation, File srcFile, String destFileName, InputStream inputStream, FileBackedOutputStream fileBackedOutputStream, StorageObjectMetadata meta, String stageRegion, String presignedUrl) Upload a file (-stream) to remote storage with Pre-signed URL without JDBC connection.

Method Detail

getMaxRetries

int getMaxRetries()

Returns:

Returns the Max number of retry attempts

getRetryBackoffMaxExponent

int getRetryBackoffMaxExponent()

Returns the max exponent for multiplying backoff with the power of 2, the value of 4 will give us 16secs as the max number of time to sleep before retry

Returns:

Returns the exponent

getRetryBackoffMin

int getRetryBackoffMin()

Returns:

Returns the min number of milliseconds to sleep before retry

isEncrypting

boolean isEncrypting()

Returns:

Returns true if encryption is enabled

getEncryptionKeySize

int getEncryptionKeySize()

Returns:

Returns the size of the encryption key

requirePresignedUrl

default boolean requirePresignedUrl()

Returns:

Whether this client requires the use of presigned URLs for upload and download instead of credentials that work for all files uploaded/ downloaded to a stage path. True for GCS.

renew

void renew(Map<?,?> stageCredentials)
    throws SnowflakeSQLException

Re-creates the encapsulated storage client with a fresh access token

Parameters:

stageCredentials - a Map (as returned by GS) which contains the new credential properties

Throws:

SnowflakeSQLException - failure to renew the storage client

shutdown

void shutdown()

shuts down the client

listObjects

StorageObjectSummaryCollection listObjects(String remoteStorageLocation,
                                           String prefix)
                                    throws StorageProviderException

For a set of remote storage objects under a remote location and a given prefix/path returns their properties wrapped in ObjectSummary objects

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix to list

Returns:

a collection of storage summary objects

Throws:

StorageProviderException - cloud storage provider error

getObjectMetadata

StorageObjectMetadata getObjectMetadata(String remoteStorageLocation,
                                        String prefix)
                                 throws StorageProviderException

Returns the metadata properties for a remote storage object

Parameters:

remoteStorageLocation - location, i.e. bucket for S3

prefix - the prefix/path of the object to retrieve

Returns:

storage metadata object

Throws:

StorageProviderException - cloud storage provider error

download

void download(SFSession connection,
              String command,
              String localLocation,
              String destFileName,
              int parallelism,
              String remoteStorageLocation,
              String stageFilePath,
              String stageRegion,
              String presignedUrl)
       throws SnowflakeSQLException

Download a file from remote storage.

Parameters:

connection - connection object

command - command to download file

localLocation - local file path

destFileName - destination file name

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for S3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - presigned URL for download. Used by GCP.

Throws:

SnowflakeSQLException - download failure

downloadToStream

InputStream downloadToStream(SFSession connection,
                             String command,
                             int parallelism,
                             String remoteStorageLocation,
                             String stageFilePath,
                             String stageRegion,
                             String presignedUrl)
                      throws SnowflakeSQLException

Download a file from remote storage

Parameters:

connection - connection object

command - command to download file

parallelism - number of threads for parallel downloading

remoteStorageLocation - remote storage location, i.e. bucket for s3

stageFilePath - stage file path

stageRegion - region name where the stage persists

presignedUrl - presigned URL for download. Used by GCP.

Returns:

input file stream

Throws:

SnowflakeSQLException - when download failure

upload

void upload(SFSession connection,
            String command,
            int parallelism,
            boolean uploadFromStream,
            String remoteStorageLocation,
            File srcFile,
            String destFileName,
            InputStream inputStream,
            FileBackedOutputStream fileBackedOutputStream,
            StorageObjectMetadata meta,
            String stageRegion,
            String presignedUrl)
     throws SnowflakeSQLException

Upload a file (-stream) to remote storage

Parameters:

connection - connection object

command - upload command

parallelism - number of threads do parallel uploading

uploadFromStream - true if upload source is stream

remoteStorageLocation - s3 bucket name

srcFile - source file if not uploading from a stream

destFileName - file name on remote storage after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - presigned URL for upload. Used by GCP.

Throws:

SnowflakeSQLException - if upload failed even after retry

uploadWithPresignedUrlWithoutConnection

default void uploadWithPresignedUrlWithoutConnection(int networkTimeoutInMilli,
                                                     HttpClientSettingsKey ocspModeAndProxyKey,
                                                     int parallelism,
                                                     boolean uploadFromStream,
                                                     String remoteStorageLocation,
                                                     File srcFile,
                                                     String destFileName,
                                                     InputStream inputStream,
                                                     FileBackedOutputStream fileBackedOutputStream,
                                                     StorageObjectMetadata meta,
                                                     String stageRegion,
                                                     String presignedUrl)
                                              throws SnowflakeSQLException

Upload a file (-stream) to remote storage with Pre-signed URL without JDBC connection.

NOTE: This function is only supported when pre-signed URL is used.

Parameters:

networkTimeoutInMilli - Network timeout for the upload

ocspModeAndProxyKey - OCSP mode and proxy settings for the upload.

parallelism - number of threads do parallel uploading

uploadFromStream - true if upload source is stream

remoteStorageLocation - s3 bucket name

srcFile - source file if not uploading from a stream

destFileName - file name on remote storage after upload

inputStream - stream used for uploading if fileBackedOutputStream is null

fileBackedOutputStream - stream used for uploading if not null

meta - object meta data

stageRegion - region name where the stage persists

presignedUrl - presigned URL for upload. Used by GCP.

Throws:

SnowflakeSQLException - if upload failed even after retry

handleStorageException

void handleStorageException(Exception ex,
                            int retryCount,
                            String operation,
                            SFSession connection,
                            String command)
                     throws SnowflakeSQLException

Handles exceptions thrown by the remote storage provider

Parameters:

ex - the exception to handle

retryCount - current number of retries, incremented by the caller before each call

operation - string that indicates the function/operation that was taking place, when the exception was raised, for example "upload"

connection - the current SFSession object used by the client

command - the command attempted at the time of the exception

Throws:

SnowflakeSQLException - exceptions that were not handled, or retried past what the retry policy allows, are propagated

getMatdescKey

String getMatdescKey()

Returns the material descriptor key

Returns:

the material descriptor key

addEncryptionMetadata

void addEncryptionMetadata(StorageObjectMetadata meta,
                           MatDesc matDesc,
                           byte[] ivData,
                           byte[] encKeK,
                           long contentLength)

Adds encryption metadata to the StorageObjectMetadata object

Parameters:

meta - the storage metadata object to add the encyption info to

matDesc - the material decriptor

ivData - the initialization vector

encKeK - the key encryption key

contentLength - the length of the encrypted content

addDigestMetadata

void addDigestMetadata(StorageObjectMetadata meta,
                       String digest)

Adds digest metadata to the StorageObjectMetadata object

Parameters:

meta - the storage metadata object to add the digest to

digest - the digest metadata to add

getDigestMetadata

String getDigestMetadata(StorageObjectMetadata meta)

Gets digest metadata to the StorageObjectMetadata object

Parameters:

meta - the metadata object to extract the digest metadata from

Returns:

the digest metadata value

addStreamingIngestMetadata

void addStreamingIngestMetadata(StorageObjectMetadata meta,
                                String clientName,
                                String clientKey)

Adds streaming ingest metadata to the StorageObjectMetadata object, used for streaming ingest per client billing calculation

Parameters:

meta - the storage metadata object to add the digest to

clientName - streaming ingest client name

clientKey - streaming ingest client key, provided by Snowflake

getStreamingIngestClientName

String getStreamingIngestClientName(StorageObjectMetadata meta)

Gets streaming ingest client name to the StorageObjectMetadata object

getStreamingIngestClientKey

String getStreamingIngestClientKey(StorageObjectMetadata meta)

Gets streaming ingest client key to the StorageObjectMetadata object