Package org.apache.beam.sdk.io.gcp.testing

Class FakeDatasetService

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams​(java.lang.String tableUrn, java.lang.Iterable<java.lang.String> writeStreamNames)
      Commit write streams of type PENDING.
      void createDataset​(java.lang.String projectId, java.lang.String datasetId, java.lang.String location, java.lang.String description, java.lang.Long defaultTableExpirationMs)
      Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
      void createTable​(com.google.api.services.bigquery.model.Table table)
      Creates the specified table if it does not exist.
      com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream​(java.lang.String tableUrn, com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
      Create a Write Stream for use with the Storage Write API.
      void deleteDataset​(java.lang.String projectId, java.lang.String datasetId)
      Deletes the dataset specified by the datasetId value.
      void deleteTable​(com.google.api.services.bigquery.model.TableReference tableRef)
      Deletes the table specified by tableId from the dataset.
      void failOnInsert​(java.util.Map<com.google.api.services.bigquery.model.TableRow,​java.util.List<com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors>> insertErrors)
      Cause a given TableRow object to fail when it's inserted.
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FinalizeWriteStreamResponse> finalizeWriteStream​(java.lang.String streamName)
      Finalize a write stream.
      com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush​(java.lang.String streamName, long offset)
      Flush a given stream up to the given offset.
      java.util.List<java.lang.String> getAllIds​(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId)  
      java.util.List<com.google.api.services.bigquery.model.TableRow> getAllRows​(java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId)  
      com.google.api.services.bigquery.model.Dataset getDataset​(java.lang.String projectId, java.lang.String datasetId)
      Gets the specified Dataset resource by dataset ID.
      int getInsertCount()  
      BigQueryServices.StreamAppendClient getStreamAppendClient​(java.lang.String streamName, com.google.protobuf.DescriptorProtos.DescriptorProto descriptor, boolean useConnectionPool, com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)
      Create an append client for a given Storage API write stream.
      com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef)
      Gets the specified Table resource by table ID.
      com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields)  
      com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)  
      com.google.api.services.bigquery.model.Table getTableImpl​(com.google.api.services.bigquery.model.TableReference tableRef, java.util.List<java.lang.String> selectedFields, BigQueryServices.DatasetService.TableMetadataView view)  
      com.google.cloud.bigquery.storage.v1.TableSchema getWriteStreamSchema​(java.lang.String streamName)  
      long insertAll​(com.google.api.services.bigquery.model.TableReference ref, java.util.List<com.google.api.services.bigquery.model.TableRow> rowList, java.util.List<java.lang.String> insertIdList)  
      <T> long insertAll​(com.google.api.services.bigquery.model.TableReference ref, java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,​com.google.api.services.bigquery.model.TableRow>> rowList, java.util.List<java.lang.String> insertIdList, InsertRetryPolicy retryPolicy, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts, ErrorContainer<T> errorContainer, boolean skipInvalidRows, boolean ignoreUnknownValues, boolean ignoreInsertIds, java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows)
      Inserts TableRows with the specified insertIds if not null.
      boolean isTableEmpty​(com.google.api.services.bigquery.model.TableReference tableRef)
      Returns true if the table is empty.
      com.google.api.services.bigquery.model.Table patchTableDescription​(com.google.api.services.bigquery.model.TableReference tableReference, java.lang.String tableDescription)
      Patch BigQuery Table description.
      void setPrimaryKey​(com.google.api.services.bigquery.model.TableReference tableReference, java.util.List<java.lang.String> columns)  
      void setShouldFailRow​(java.util.function.Function<com.google.api.services.bigquery.model.TableRow,​java.lang.Boolean> shouldFailRow)  
      static void setUp()  
      void updateTableSchema​(com.google.api.services.bigquery.model.TableReference tableReference, com.google.api.services.bigquery.model.TableSchema tableSchema)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • FakeDatasetService

        public FakeDatasetService()

    • Method Detail

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.lang.Exception

      • setUp

        public static void setUp()

      • setShouldFailRow

        public void setShouldFailRow​(java.util.function.Function<com.google.api.services.bigquery.model.TableRow,​java.lang.Boolean> shouldFailRow)

      • getTable

        public com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef)
                                                      throws java.lang.InterruptedException,
                                                             java.io.IOException
        Description copied from interface: BigQueryServices.DatasetService
        Gets the specified Table resource by table ID.

        Returns null if the table is not found.

        Specified by:
        getTable in interface BigQueryServices.DatasetService
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getTable

        public com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef,
                                                             @Nullable
                                                             java.util.List<java.lang.String> selectedFields)
                                                      throws java.lang.InterruptedException,
                                                             java.io.IOException
        Specified by:
        getTable in interface BigQueryServices.DatasetService
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getTable

        public com.google.api.services.bigquery.model.Table getTable​(com.google.api.services.bigquery.model.TableReference tableRef,
                                                             @Nullable
                                                             java.util.List<java.lang.String> selectedFields,
                                                             @Nullable
                                                             BigQueryServices.DatasetService.TableMetadataView view)
                                                      throws java.lang.InterruptedException,
                                                             java.io.IOException
        Specified by:
        getTable in interface BigQueryServices.DatasetService
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getTableImpl

        public com.google.api.services.bigquery.model.Table getTableImpl​(com.google.api.services.bigquery.model.TableReference tableRef,
                                                                 @Nullable
                                                                 java.util.List<java.lang.String> selectedFields,
                                                                 @Nullable
                                                                 BigQueryServices.DatasetService.TableMetadataView view)
                                                          throws java.lang.InterruptedException,
                                                                 java.io.IOException
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getAllRows

        public java.util.List<com.google.api.services.bigquery.model.TableRow> getAllRows​(java.lang.String projectId,
                                                                                  java.lang.String datasetId,
                                                                                  java.lang.String tableId)
                                                                           throws java.lang.InterruptedException,
                                                                                  java.io.IOException
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • getAllIds

        public java.util.List<java.lang.String> getAllIds​(java.lang.String projectId,
                                                  java.lang.String datasetId,
                                                  java.lang.String tableId)
                                           throws java.lang.InterruptedException,
                                                  java.io.IOException
        Throws:
        java.lang.InterruptedException
        java.io.IOException

      • deleteTable

        public void deleteTable​(com.google.api.services.bigquery.model.TableReference tableRef)
                 throws java.io.IOException,
                        java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Deletes the table specified by tableId from the dataset. If the table contains data, all the data will be deleted.
        Specified by:
        deleteTable in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • updateTableSchema

        public void updateTableSchema​(com.google.api.services.bigquery.model.TableReference tableReference,
                              com.google.api.services.bigquery.model.TableSchema tableSchema)
                       throws java.io.IOException
        Throws:
        java.io.IOException

      • setPrimaryKey

        public void setPrimaryKey​(com.google.api.services.bigquery.model.TableReference tableReference,
                          java.util.List<java.lang.String> columns)
                   throws java.io.IOException
        Throws:
        java.io.IOException

      • isTableEmpty

        public boolean isTableEmpty​(com.google.api.services.bigquery.model.TableReference tableRef)
                     throws java.io.IOException,
                            java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Returns true if the table is empty.
        Specified by:
        isTableEmpty in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException - if the table is not found.
        java.lang.InterruptedException

      • getDataset

        public com.google.api.services.bigquery.model.Dataset getDataset​(java.lang.String projectId,
                                                                 java.lang.String datasetId)
                                                          throws java.io.IOException,
                                                                 java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Gets the specified Dataset resource by dataset ID.
        Specified by:
        getDataset in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • createDataset

        public void createDataset​(java.lang.String projectId,
                          java.lang.String datasetId,
                          java.lang.String location,
                          java.lang.String description,
                          java.lang.Long defaultTableExpirationMs)
                   throws java.io.IOException,
                          java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Create a Dataset with the given location, description and default expiration time for tables in the dataset (if null, tables don't expire).
        Specified by:
        createDataset in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • deleteDataset

        public void deleteDataset​(java.lang.String projectId,
                          java.lang.String datasetId)
                   throws java.io.IOException,
                          java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Deletes the dataset specified by the datasetId value.

        Before you can delete a dataset, you must delete all its tables.

        Specified by:
        deleteDataset in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • getInsertCount

        public int getInsertCount()

      • insertAll

        public long insertAll​(com.google.api.services.bigquery.model.TableReference ref,
                      java.util.List<com.google.api.services.bigquery.model.TableRow> rowList,
                      @Nullable
                      java.util.List<java.lang.String> insertIdList)
               throws java.io.IOException,
                      java.lang.InterruptedException
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • insertAll

        public <T> long insertAll​(com.google.api.services.bigquery.model.TableReference ref,
                          java.util.List<org.apache.beam.sdk.values.FailsafeValueInSingleWindow<com.google.api.services.bigquery.model.TableRow,​com.google.api.services.bigquery.model.TableRow>> rowList,
                          @Nullable
                          java.util.List<java.lang.String> insertIdList,
                          InsertRetryPolicy retryPolicy,
                          java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<T>> failedInserts,
                          ErrorContainer<T> errorContainer,
                          boolean skipInvalidRows,
                          boolean ignoreUnknownValues,
                          boolean ignoreInsertIds,
                          java.util.List<org.apache.beam.sdk.values.ValueInSingleWindow<com.google.api.services.bigquery.model.TableRow>> successfulRows)
                   throws java.io.IOException,
                          java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Inserts TableRows with the specified insertIds if not null.

        If any insert fail permanently according to the retry policy, those rows are added to failedInserts.

        Returns the total bytes count of TableRows.

        Specified by:
        insertAll in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • patchTableDescription

        public com.google.api.services.bigquery.model.Table patchTableDescription​(com.google.api.services.bigquery.model.TableReference tableReference,
                                                                          @Nullable
                                                                          java.lang.String tableDescription)
                                                                   throws java.io.IOException,
                                                                          java.lang.InterruptedException
        Description copied from interface: BigQueryServices.DatasetService
        Patch BigQuery Table description.
        Specified by:
        patchTableDescription in interface BigQueryServices.DatasetService
        Throws:
        java.io.IOException
        java.lang.InterruptedException

      • createWriteStream

        public com.google.cloud.bigquery.storage.v1.WriteStream createWriteStream​(java.lang.String tableUrn,
                                                                          com.google.cloud.bigquery.storage.v1.WriteStream.Type type)
                                                                   throws java.lang.InterruptedException
        Description copied from interface: BigQueryServices.WriteStreamService
        Create a Write Stream for use with the Storage Write API.
        Specified by:
        createWriteStream in interface BigQueryServices.WriteStreamService
        Throws:
        java.lang.InterruptedException

      • getStreamAppendClient

        public BigQueryServices.StreamAppendClient getStreamAppendClient​(java.lang.String streamName,
                                                                 com.google.protobuf.DescriptorProtos.DescriptorProto descriptor,
                                                                 boolean useConnectionPool,
                                                                 com.google.cloud.bigquery.storage.v1.AppendRowsRequest.MissingValueInterpretation missingValueInterpretation)
                                                          throws java.lang.Exception
        Description copied from interface: BigQueryServices.WriteStreamService
        Create an append client for a given Storage API write stream. The stream must be created first.
        Specified by:
        getStreamAppendClient in interface BigQueryServices.WriteStreamService
        Throws:
        java.lang.Exception

      • flush

        public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.FlushRowsResponse> flush​(java.lang.String streamName,
                                                                                                   long offset)
        Description copied from interface: BigQueryServices.WriteStreamService
        Flush a given stream up to the given offset. The stream must have type BUFFERED.
        Specified by:
        flush in interface BigQueryServices.WriteStreamService

      • commitWriteStreams

        public com.google.api.core.ApiFuture<com.google.cloud.bigquery.storage.v1.BatchCommitWriteStreamsResponse> commitWriteStreams​(java.lang.String tableUrn,
                                                                                                                              java.lang.Iterable<java.lang.String> writeStreamNames)
        Description copied from interface: BigQueryServices.WriteStreamService
        Commit write streams of type PENDING. The streams must be finalized before committing.
        Specified by:
        commitWriteStreams in interface BigQueryServices.WriteStreamService

      • failOnInsert

        public void failOnInsert​(java.util.Map<com.google.api.services.bigquery.model.TableRow,​java.util.List<com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors>> insertErrors)
        Cause a given TableRow object to fail when it's inserted. The errors link the list will be returned on subsequent retries, and the insert will succeed when the errors run out.