Package org.apache.beam.runners.dataflow.internal

Class IsmFormat.IsmRecordCoder<V>

  • All Implemented Interfaces:
    java.io.Serializable
    Enclosing class:
    IsmFormat
    public static class IsmFormat.IsmRecordCoder<V>
extends org.apache.beam.sdk.coders.CustomCoder<IsmFormat.IsmRecord<V>>
    A Coder for IsmFormat.IsmRecords.

    Note that this coder standalone will not produce an Ism file. This coder can be used to materialize a PCollection of IsmFormat.IsmRecords. Only when this coder is combined with an IsmSink will one produce an Ism file.

    The IsmFormat.IsmRecord encoded format is:

    • encoded key component 1 using key component coder 1
    • ...
    • encoded key component N using key component coder N
    • encoded value using value coder
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.beam.sdk.coders.Coder

        org.apache.beam.sdk.coders.Coder.Context, org.apache.beam.sdk.coders.Coder.NonDeterministicException

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      boolean consistentWithEquals()  
      IsmFormat.IsmRecord<V> decode​(java.io.InputStream inStream)  
      void encode​(IsmFormat.IsmRecord<V> value, java.io.OutputStream outStream)  
      int encodeAndHash​(java.util.List<?> keyComponents, RandomAccessData keyBytesToMutate)
      Computes the shard id for the given key component(s).
      int encodeAndHash​(java.util.List<?> keyComponents, RandomAccessData keyBytesToMutate, java.util.List<java.lang.Integer> keyComponentByteOffsetsToMutate)
      Computes the shard id for the given key component(s).
      boolean equals​(@Nullable java.lang.Object other)  
      java.util.List<org.apache.beam.sdk.coders.Coder<?>> getCoderArguments()  
      <T> org.apache.beam.sdk.coders.Coder<T> getKeyComponentCoder​(int index)
      Returns the key coder at the specified index.
      java.util.List<org.apache.beam.sdk.coders.Coder<?>> getKeyComponentCoders()
      Returns the list of key component coders.
      int getNumberOfShardKeyCoders​(java.util.List<?> keyComponents)  
      org.apache.beam.sdk.coders.Coder<V> getValueCoder()
      Returns the value coder.
      int hash​(java.util.List<?> keyComponents)
      Computes the shard id for the given key component(s).
      int hashCode()  
      static <V> IsmFormat.IsmRecordCoder<V> of​(int numberOfShardKeyCoders, int numberOfMetadataShardKeyCoders, java.util.List<org.apache.beam.sdk.coders.Coder<?>> keyComponentCoders, org.apache.beam.sdk.coders.Coder<V> valueCoder)
      Returns an IsmRecordCoder with the specified key component coders, value coder.
      java.lang.Object structuralValue​(IsmFormat.IsmRecord<V> record)  
      void verifyDeterministic()  

      • Methods inherited from class org.apache.beam.sdk.coders.Coder

        decode, encode, getEncodedElementByteSize, getEncodedElementByteSizeUsingCoder, getEncodedTypeDescriptor, isRegisterByteSizeObserverCheap, registerByteSizeObserver, verifyDeterministic, verifyDeterministic

      • Methods inherited from class java.lang.Object

        clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • of

        public static <V> IsmFormat.IsmRecordCoder<V> of​(int numberOfShardKeyCoders,
                                                 int numberOfMetadataShardKeyCoders,
                                                 java.util.List<org.apache.beam.sdk.coders.Coder<?>> keyComponentCoders,
                                                 org.apache.beam.sdk.coders.Coder<V> valueCoder)
        Returns an IsmRecordCoder with the specified key component coders, value coder.

      • getKeyComponentCoders

        public java.util.List<org.apache.beam.sdk.coders.Coder<?>> getKeyComponentCoders()
        Returns the list of key component coders.

      • getKeyComponentCoder

        public <T> org.apache.beam.sdk.coders.Coder<T> getKeyComponentCoder​(int index)
        Returns the key coder at the specified index.

      • getValueCoder

        public org.apache.beam.sdk.coders.Coder<V> getValueCoder()
        Returns the value coder.

      • encode

        public void encode​(IsmFormat.IsmRecord<V> value,
                   java.io.OutputStream outStream)
            throws org.apache.beam.sdk.coders.CoderException,
                   java.io.IOException
        Specified by:
        encode in class org.apache.beam.sdk.coders.Coder<IsmFormat.IsmRecord<V>>
        Throws:
        org.apache.beam.sdk.coders.CoderException
        java.io.IOException

      • decode

        public IsmFormat.IsmRecord<V> decode​(java.io.InputStream inStream)
                              throws org.apache.beam.sdk.coders.CoderException,
                                     java.io.IOException
        Specified by:
        decode in class org.apache.beam.sdk.coders.Coder<IsmFormat.IsmRecord<V>>
        Throws:
        org.apache.beam.sdk.coders.CoderException
        java.io.IOException

      • getNumberOfShardKeyCoders

        public int getNumberOfShardKeyCoders​(java.util.List<?> keyComponents)

      • hash

        public int hash​(java.util.List<?> keyComponents)
        Computes the shard id for the given key component(s).

        The shard keys are encoded into their byte representations and hashed using the 32-bit murmur3 algorithm, x86 variant (little-endian variant), using 1225801234 as the seed value. We ensure that shard ids for metadata keys and normal keys do not overlap.

      • encodeAndHash

        public int encodeAndHash​(java.util.List<?> keyComponents,
                         RandomAccessData keyBytesToMutate)
        Computes the shard id for the given key component(s).

        Mutates keyBytes such that when returned, contains the encoded version of the key components.

      • encodeAndHash

        public int encodeAndHash​(java.util.List<?> keyComponents,
                         RandomAccessData keyBytesToMutate,
                         java.util.List<java.lang.Integer> keyComponentByteOffsetsToMutate)
        Computes the shard id for the given key component(s).

        Mutates keyBytes such that when returned, contains the encoded version of the key components. Also, mutates keyComponentByteOffsetsToMutate to store the location where each key component's encoded byte representation ends within keyBytes.

      • getCoderArguments

        public java.util.List<org.apache.beam.sdk.coders.Coder<?>> getCoderArguments()
        Overrides:
        getCoderArguments in class org.apache.beam.sdk.coders.CustomCoder<IsmFormat.IsmRecord<V>>

      • verifyDeterministic

        public void verifyDeterministic()
                         throws org.apache.beam.sdk.coders.Coder.NonDeterministicException
        Overrides:
        verifyDeterministic in class org.apache.beam.sdk.coders.CustomCoder<IsmFormat.IsmRecord<V>>
        Throws:
        org.apache.beam.sdk.coders.Coder.NonDeterministicException

      • consistentWithEquals

        public boolean consistentWithEquals()
        Overrides:
        consistentWithEquals in class org.apache.beam.sdk.coders.Coder<IsmFormat.IsmRecord<V>>

      • equals

        public boolean equals​(@Nullable java.lang.Object other)
        Overrides:
        equals in class java.lang.Object

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class java.lang.Object