Package org.apache.arrow.memory

Class ArrowBuf

  • All Implemented Interfaces:
    AutoCloseable
    Direct Known Subclasses:
    DatabricksArrowBuf
    public class ArrowBuf
extends Object
implements AutoCloseable
    ArrowBuf serves as a facade over underlying memory by providing several access APIs to read/write data into a chunk of direct memory. All the accounting, ownership and reference management is done by ReferenceManager and ArrowBuf can work with a custom user provided implementation of ReferenceManager

    Two important instance variables of an ArrowBuf: (1) address - starting virtual address in the underlying memory chunk that this ArrowBuf has access to (2) length - length (in bytes) in the underlying memory chunk that this ArrowBuf has access to

    The management (allocation, deallocation, reference counting etc) for the memory chunk is not done by ArrowBuf. Default implementation of ReferenceManager, allocation is in BaseAllocator, BufferLedger and AllocationManager

    • Constructor Summary

      Constructors 
      Constructor Description
      ArrowBuf​(org.apache.arrow.memory.ReferenceManager referenceManager, org.apache.arrow.memory.BufferManager bufferManager, long capacity, long memoryAddress)
      Constructs a new ArrowBuf.

    • Method Summary

      All Methods Instance Methods Concrete Methods Deprecated Methods 
      Modifier and Type Method Description
      long capacity()  
      ArrowBuf capacity​(long newCapacity)
      Adjusts the capacity of this buffer.
      void checkBytes​(long start, long end)
      Allows a function to determine whether not reading a particular string of bytes is valid.
      ArrowBuf clear()  
      void close()  
      boolean equals​(Object obj)  
      long getActualMemoryConsumed()
      Return that is Accounted for by this buffer (and its potentially shared siblings within the context of the associated allocator).
      byte getByte​(long index)
      Get byte value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void getBytes​(long index, byte[] dst)
      Copy data from this ArrowBuf at a given index in into destination byte array.
      void getBytes​(long index, byte[] dst, int dstIndex, int length)
      Copy data from this ArrowBuf at a given index into destination byte array.
      void getBytes​(long index, OutputStream out, int length)
      Copy a certain length of bytes from this ArrowBuf at a given index into the given OutputStream.
      void getBytes​(long index, ByteBuffer dst)
      Copy data from this ArrowBuf at a given index into the destination ByteBuffer.
      void getBytes​(long index, ArrowBuf dst, long dstIndex, int length)
      Copy a given length of data from this ArrowBuf starting at a given index into a dst ArrowBuf at dstIndex.
      char getChar​(long index)
      Get char value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      double getDouble​(long index)
      Get double value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      float getFloat​(long index)
      Get float value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      long getId()
      Get the integer id assigned to this ArrowBuf for debugging purposes.
      int getInt​(long index)
      Get int value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      long getLong​(long index)
      Get long value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      long getPossibleMemoryConsumed()
      Returns the possible memory consumed by this ArrowBuf in the worse case scenario.
      org.apache.arrow.memory.ReferenceManager getReferenceManager()
      Get reference manager for this ArrowBuf.
      short getShort​(long index)
      Get short value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
      int hashCode()  
      long memoryAddress()  
      ByteBuffer nioBuffer()
      Make a nio byte buffer from this arrowbuf.
      ByteBuffer nioBuffer​(long index, int length)
      Make a nio byte buffer from this ArrowBuf.
      ByteOrder order()
      Returns the byte order of elements in this buffer.
      void print​(StringBuilder sb, int indent)
      Print detailed information of this buffer into sb.
      long readableBytes()
      Returns the number of bytes still available to read in this buffer.
      byte readByte()
      Read the byte at readerIndex.
      void readBytes​(byte[] dst)
      Read dst.length bytes at readerIndex into dst byte array.
      long readerIndex()
      Get the index at which the next byte will be read from.
      ArrowBuf readerIndex​(long readerIndex)
      Set the reader index for this ArrowBuf.
      ArrowBuf reallocIfNeeded​(long size)
      Returns this if size is less than capacity(), otherwise delegates to BufferManager.replace(ArrowBuf, long) to get a new buffer.
      int refCnt()  
      void setByte​(long index, byte value)
      Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setByte​(long index, int value)
      Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setBytes​(long index, byte[] src)
      Copy data from a given byte array into this ArrowBuf starting at a given index.
      void setBytes​(long index, byte[] src, int srcIndex, long length)
      Copy data from a given byte array starting at the given source index into this ArrowBuf at a given index.
      int setBytes​(long index, InputStream in, int length)
      Copy a certain length of bytes from given InputStream into this ArrowBuf at the provided index.
      void setBytes​(long index, ByteBuffer src)
      Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer.
      void setBytes​(long index, ByteBuffer src, int srcIndex, int length)
      Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer starting at a given srcIndex for a certain length.
      void setBytes​(long index, ArrowBuf src)
      Copy readableBytes() number of bytes from src ArrowBuf starting from its readerIndex into this ArrowBuf starting at the given index.
      void setBytes​(long index, ArrowBuf src, long srcIndex, long length)
      Copy data from src ArrowBuf starting at index srcIndex into this ArrowBuf at given index.
      void setChar​(long index, int value)
      Set char value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setDouble​(long index, double value)
      Set double value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setFloat​(long index, float value)
      Set float value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setInt​(long index, int value)
      Set int value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setLong​(long index, long value)
      Set long value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      ArrowBuf setOne​(int index, int length)
      Deprecated.
      use setOne(long, long) instead.
      ArrowBuf setOne​(long index, long length)
      Sets all bits to one in the specified range.
      void setShort​(long index, int value)
      Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      void setShort​(long index, short value)
      Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
      ArrowBuf setZero​(long index, long length)
      Zero-out the bytes in this ArrowBuf starting at the given index for the given length.
      ArrowBuf slice()
      Returns a slice of only the readable bytes in the buffer.
      ArrowBuf slice​(long index, long length)
      Returns a slice (view) starting at index with the given length.
      String toHexString​(long start, int length)
      Return the buffer's byte contents in the form of a hex dump.
      String toString()  
      long writableBytes()
      Returns the number of bytes still available to write into this buffer before capacity is reached.
      void writeByte​(byte value)
      Set the provided byte value at the writerIndex.
      void writeByte​(int value)
      Set the lower order byte for the provided value at the writerIndex.
      void writeBytes​(byte[] src)
      Write the bytes from given byte array into this ArrowBuf starting at writerIndex.
      void writeBytes​(byte[] src, int srcIndex, int length)
      Write the bytes from given byte array starting at srcIndex into this ArrowBuf starting at writerIndex.
      void writeDouble​(double value)
      Set the provided double value at the writerIndex.
      void writeFloat​(float value)
      Set the provided float value at the writerIndex.
      void writeInt​(int value)
      Set the provided int value at the writerIndex.
      void writeLong​(long value)
      Set the provided long value at the writerIndex.
      long writerIndex()
      Get the index at which next byte will be written to.
      ArrowBuf writerIndex​(long writerIndex)
      Set the writer index for this ArrowBuf.
      void writeShort​(int value)
      Set the provided int value as short at the writerIndex.

    • Constructor Detail

      • ArrowBuf

        public ArrowBuf​(org.apache.arrow.memory.ReferenceManager referenceManager,
                org.apache.arrow.memory.BufferManager bufferManager,
                long capacity,
                long memoryAddress)
        Constructs a new ArrowBuf.
        Parameters:
        referenceManager - The memory manager to track memory usage and reference count of this buffer
        capacity - The capacity in bytes of this buffer

    • Method Detail

      • refCnt

        public int refCnt()

      • checkBytes

        public void checkBytes​(long start,
                       long end)
        Allows a function to determine whether not reading a particular string of bytes is valid.

        Will throw an exception if the memory is not readable for some reason. Only doesn't something in the case that AssertionUtil.BOUNDS_CHECKING_ENABLED is true.

        Parameters:
        start - The starting position of the bytes to be read.
        end - The exclusive endpoint of the bytes to be read.

      • getReferenceManager

        public org.apache.arrow.memory.ReferenceManager getReferenceManager()
        Get reference manager for this ArrowBuf.
        Returns:
        user provided implementation of ReferenceManager

      • capacity

        public long capacity()

      • capacity

        public ArrowBuf capacity​(long newCapacity)
        Adjusts the capacity of this buffer. Size increases are NOT supported.
        Parameters:
        newCapacity - Must be in in the range [0, length).

      • order

        public ByteOrder order()
        Returns the byte order of elements in this buffer.

      • readableBytes

        public long readableBytes()
        Returns the number of bytes still available to read in this buffer.

      • writableBytes

        public long writableBytes()
        Returns the number of bytes still available to write into this buffer before capacity is reached.

      • slice

        public ArrowBuf slice()
        Returns a slice of only the readable bytes in the buffer.

      • slice

        public ArrowBuf slice​(long index,
                      long length)
        Returns a slice (view) starting at index with the given length.

      • nioBuffer

        public ByteBuffer nioBuffer()
        Make a nio byte buffer from this arrowbuf.

      • nioBuffer

        public ByteBuffer nioBuffer​(long index,
                            int length)
        Make a nio byte buffer from this ArrowBuf.

      • memoryAddress

        public long memoryAddress()

      • hashCode

        public int hashCode()
        Overrides:
        hashCode in class Object

      • getLong

        public long getLong​(long index)
        Get long value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        8 byte long value

      • setLong

        public void setLong​(long index,
                    long value)
        Set long value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getFloat

        public float getFloat​(long index)
        Get float value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        4 byte float value

      • setFloat

        public void setFloat​(long index,
                     float value)
        Set float value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getDouble

        public double getDouble​(long index)
        Get double value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        8 byte double value

      • setDouble

        public void setDouble​(long index,
                      double value)
        Set double value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getChar

        public char getChar​(long index)
        Get char value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        2 byte char value

      • setChar

        public void setChar​(long index,
                    int value)
        Set char value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getInt

        public int getInt​(long index)
        Get int value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        4 byte int value

      • setInt

        public void setInt​(long index,
                   int value)
        Set int value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getShort

        public short getShort​(long index)
        Get short value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        2 byte short value

      • setShort

        public void setShort​(long index,
                     int value)
        Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • setShort

        public void setShort​(long index,
                     short value)
        Set short value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • setByte

        public void setByte​(long index,
                    int value)
        Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • setByte

        public void setByte​(long index,
                    byte value)
        Set byte value at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be written
        value - value to write

      • getByte

        public byte getByte​(long index)
        Get byte value stored at a particular index in the underlying memory chunk this ArrowBuf has access to.
        Parameters:
        index - index (0 based relative to this ArrowBuf) where the value will be read from
        Returns:
        byte value

      • readByte

        public byte readByte()
        Read the byte at readerIndex.
        Returns:
        byte value

      • readBytes

        public void readBytes​(byte[] dst)
        Read dst.length bytes at readerIndex into dst byte array.
        Parameters:
        dst - byte array where the data will be written

      • writeByte

        public void writeByte​(byte value)
        Set the provided byte value at the writerIndex.
        Parameters:
        value - value to set

      • writeByte

        public void writeByte​(int value)
        Set the lower order byte for the provided value at the writerIndex.
        Parameters:
        value - value to be set

      • writeBytes

        public void writeBytes​(byte[] src)
        Write the bytes from given byte array into this ArrowBuf starting at writerIndex.
        Parameters:
        src - src byte array

      • writeBytes

        public void writeBytes​(byte[] src,
                       int srcIndex,
                       int length)
        Write the bytes from given byte array starting at srcIndex into this ArrowBuf starting at writerIndex.
        Parameters:
        src - src byte array
        srcIndex - index in the byte array where the copy will being from
        length - length of data to copy

      • writeShort

        public void writeShort​(int value)
        Set the provided int value as short at the writerIndex.
        Parameters:
        value - value to set

      • writeInt

        public void writeInt​(int value)
        Set the provided int value at the writerIndex.
        Parameters:
        value - value to set

      • writeLong

        public void writeLong​(long value)
        Set the provided long value at the writerIndex.
        Parameters:
        value - value to set

      • writeFloat

        public void writeFloat​(float value)
        Set the provided float value at the writerIndex.
        Parameters:
        value - value to set

      • writeDouble

        public void writeDouble​(double value)
        Set the provided double value at the writerIndex.
        Parameters:
        value - value to set

      • getBytes

        public void getBytes​(long index,
                     byte[] dst)
        Copy data from this ArrowBuf at a given index in into destination byte array.
        Parameters:
        index - starting index (0 based relative to the portion of memory) this ArrowBuf has access to
        dst - byte array to copy the data into

      • getBytes

        public void getBytes​(long index,
                     byte[] dst,
                     int dstIndex,
                     int length)
        Copy data from this ArrowBuf at a given index into destination byte array.
        Parameters:
        index - index (0 based relative to the portion of memory this ArrowBuf has access to)
        dst - byte array to copy the data into
        dstIndex - starting index in dst byte array to copy into
        length - length of data to copy from this ArrowBuf

      • setBytes

        public void setBytes​(long index,
                     byte[] src)
        Copy data from a given byte array into this ArrowBuf starting at a given index.
        Parameters:
        index - starting index (0 based relative to the portion of memory) this ArrowBuf has access to
        src - byte array to copy the data from

      • setBytes

        public void setBytes​(long index,
                     byte[] src,
                     int srcIndex,
                     long length)
        Copy data from a given byte array starting at the given source index into this ArrowBuf at a given index.
        Parameters:
        index - index (0 based relative to the portion of memory this ArrowBuf has access to)
        src - src byte array to copy the data from
        srcIndex - index in the byte array where the copy will start from
        length - length of data to copy from byte array

      • getBytes

        public void getBytes​(long index,
                     ByteBuffer dst)
        Copy data from this ArrowBuf at a given index into the destination ByteBuffer.
        Parameters:
        index - index (0 based relative to the portion of memory this ArrowBuf has access to)
        dst - dst ByteBuffer where the data will be copied into

      • setBytes

        public void setBytes​(long index,
                     ByteBuffer src)
        Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        src - src ByteBuffer where the data will be copied from

      • setBytes

        public void setBytes​(long index,
                     ByteBuffer src,
                     int srcIndex,
                     int length)
        Copy data into this ArrowBuf at a given index onwards from a source ByteBuffer starting at a given srcIndex for a certain length.
        Parameters:
        index - index (0 based relative to the portion of memory this ArrowBuf has access to)
        src - src ByteBuffer where the data will be copied from
        srcIndex - starting index in the src ByteBuffer where the data copy will start from
        length - length of data to copy from src ByteBuffer

      • getBytes

        public void getBytes​(long index,
                     ArrowBuf dst,
                     long dstIndex,
                     int length)
        Copy a given length of data from this ArrowBuf starting at a given index into a dst ArrowBuf at dstIndex.
        Parameters:
        index - index (0 based relative to the portion of memory this ArrowBuf has access to)
        dst - dst ArrowBuf where the data will be copied into
        dstIndex - index (0 based relative to the portion of memory dst ArrowBuf has access to)
        length - length of data to copy

      • setBytes

        public void setBytes​(long index,
                     ArrowBuf src,
                     long srcIndex,
                     long length)
        Copy data from src ArrowBuf starting at index srcIndex into this ArrowBuf at given index.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        src - src ArrowBuf where the data will be copied from
        srcIndex - starting index in the src ArrowBuf where the copy will begin from
        length - length of data to copy from src ArrowBuf

      • setBytes

        public void setBytes​(long index,
                     ArrowBuf src)
        Copy readableBytes() number of bytes from src ArrowBuf starting from its readerIndex into this ArrowBuf starting at the given index.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        src - src ArrowBuf where the data will be copied from

      • setBytes

        public int setBytes​(long index,
                    InputStream in,
                    int length)
             throws IOException
        Copy a certain length of bytes from given InputStream into this ArrowBuf at the provided index.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        in - src stream to copy from
        length - length of data to copy
        Returns:
        number of bytes copied from stream into ArrowBuf
        Throws:
        IOException - on failing to read from stream

      • getBytes

        public void getBytes​(long index,
                     OutputStream out,
                     int length)
              throws IOException
        Copy a certain length of bytes from this ArrowBuf at a given index into the given OutputStream.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        out - dst stream to copy data into
        length - length of data to copy
        Throws:
        IOException - on failing to write to stream

      • getPossibleMemoryConsumed

        public long getPossibleMemoryConsumed()
        Returns the possible memory consumed by this ArrowBuf in the worse case scenario. (not shared, connected to larger underlying buffer of allocated memory)
        Returns:
        Size in bytes.

      • getActualMemoryConsumed

        public long getActualMemoryConsumed()
        Return that is Accounted for by this buffer (and its potentially shared siblings within the context of the associated allocator).
        Returns:
        Size in bytes.

      • toHexString

        public String toHexString​(long start,
                          int length)
        Return the buffer's byte contents in the form of a hex dump.
        Parameters:
        start - the starting byte index
        length - how many bytes to log
        Returns:
        A hex dump in a String.

      • getId

        public long getId()
        Get the integer id assigned to this ArrowBuf for debugging purposes.
        Returns:
        integer id

      • print

        public void print​(StringBuilder sb,
                  int indent)
        Print detailed information of this buffer into sb.

        Most information will only be present if BaseAllocator.DEBUG is true.

      • readerIndex

        public long readerIndex()
        Get the index at which the next byte will be read from.
        Returns:
        reader index

      • writerIndex

        public long writerIndex()
        Get the index at which next byte will be written to.
        Returns:
        writer index

      • readerIndex

        public ArrowBuf readerIndex​(long readerIndex)
        Set the reader index for this ArrowBuf.
        Parameters:
        readerIndex - new reader index
        Returns:
        this ArrowBuf

      • writerIndex

        public ArrowBuf writerIndex​(long writerIndex)
        Set the writer index for this ArrowBuf.
        Parameters:
        writerIndex - new writer index
        Returns:
        this ArrowBuf

      • setZero

        public ArrowBuf setZero​(long index,
                        long length)
        Zero-out the bytes in this ArrowBuf starting at the given index for the given length.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        length - length of bytes to zero-out
        Returns:
        this ArrowBuf

      • setOne

        @Deprecated
public ArrowBuf setOne​(int index,
                       int length)
        Deprecated.
        use setOne(long, long) instead.
        Sets all bits to one in the specified range.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        length - length of bytes to set.
        Returns:
        this ArrowBuf

      • setOne

        public ArrowBuf setOne​(long index,
                       long length)
        Sets all bits to one in the specified range.
        Parameters:
        index - index index (0 based relative to the portion of memory this ArrowBuf has access to)
        length - length of bytes to set.
        Returns:
        this ArrowBuf

      • reallocIfNeeded

        public ArrowBuf reallocIfNeeded​(long size)
        Returns this if size is less than capacity(), otherwise delegates to BufferManager.replace(ArrowBuf, long) to get a new buffer.