Package org.apache.arrow.vector.complex

Class ListViewVector

java.lang.Object
org.apache.arrow.vector.BaseValueVector
org.apache.arrow.vector.complex.BaseRepeatedValueViewVector
org.apache.arrow.vector.complex.ListViewVector
All Implemented Interfaces:
Closeable, AutoCloseable, Iterable<ValueVector>, BaseListVector, PromotableVector, RepeatedValueVector, DensityAwareVector, FieldVector, ValueIterableVector<List<?>>, ValueVector
public class ListViewVector extends BaseRepeatedValueViewVector implements PromotableVector, ValueIterableVector<List<?>>
A list view vector contains lists of a specific type of elements. Its structure contains four elements.
  1. A validity buffer.
  2. An offset buffer, that denotes lists starts.
  3. A size buffer, that denotes lists ends.
  4. A child data vector that contains the elements of lists.
The latter three are managed by its superclass.

  • Field Details

    • validityBuffer

      protected ArrowBuf validityBuffer

    • reader

      protected UnionListReader reader

    • field

      protected Field field

    • validityAllocationSizeInBytes

      protected int validityAllocationSizeInBytes

  • Constructor Details

    • ListViewVector

      public ListViewVector(String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack)
      Constructs a new instance.
      Parameters:
      name - The name of the instance.
      allocator - The allocator to use for allocating/reallocating buffers.
      fieldType - The type of this list.
      callBack - A schema change callback.

    • ListViewVector

      public ListViewVector(Field field, BufferAllocator allocator, CallBack callBack)
      Constructs a new instance.
      Parameters:
      field - The field materialized by this vector.
      allocator - The allocator to use for allocating/reallocating buffers.
      callBack - A schema change callback.

  • Method Details

    • empty

      public static ListViewVector empty(String name, BufferAllocator allocator)

    • initializeChildrenFromFields

      public void initializeChildrenFromFields(List<Field> children)
      Description copied from interface: FieldVector
      Initializes the child vectors to be later loaded with loadBuffers.
      Specified by:
      initializeChildrenFromFields in interface FieldVector
      Parameters:
      children - the schema

    • setInitialCapacity

      public void setInitialCapacity(int numRecords)
      Description copied from interface: ValueVector
      Set the initial record capacity.
      Specified by:
      setInitialCapacity in interface ValueVector
      Overrides:
      setInitialCapacity in class BaseRepeatedValueViewVector
      Parameters:
      numRecords - the initial record capacity.

    • setInitialCapacity

      public void setInitialCapacity(int numRecords, double density)
      Specialized version of setInitialCapacity() for ListViewVector. This is used by some callers when they want to explicitly control and be conservative about memory allocated for inner data vector. This is very useful when we are working with memory constraints for a query and have a fixed amount of memory reserved for the record batch. In such cases, we are likely to face OOM or related problems when we reserve memory for a record batch with value count x and do setInitialCapacity(x) such that each vector allocates only what is necessary and not the default amount, but the multiplier forces the memory requirement to go beyond what was needed.
      Specified by:
      setInitialCapacity in interface DensityAwareVector
      Overrides:
      setInitialCapacity in class BaseRepeatedValueViewVector
      Parameters:
      numRecords - value count
      density - density of ListViewVector. Density is the average size of a list per position in the ListViewVector. For example, a density value of 10 implies each position in the list vector has a list of 10 values. A density value of 0.1 implies out of 10 positions in the list vector, 1 position has a list of size 1, and the remaining positions are null (no lists) or empty lists. This helps in tightly controlling the memory we provision for inner data vector.

    • setInitialTotalCapacity

      public void setInitialTotalCapacity(int numRecords, int totalNumberOfElements)
      Specialized version of setInitialTotalCapacity() for ListViewVector. This is used by some callers when they want to explicitly control and be conservative about memory allocated for inner data vector. This is very useful when we are working with memory constraints for a query and have a fixed amount of memory reserved for the record batch. In such cases, we are likely to face OOM or related problems when we reserve memory for a record batch with value count x and do setInitialCapacity(x) such that each vector allocates only what is necessary and not the default amount, but the multiplier forces the memory requirement to go beyond what was needed.
      Overrides:
      setInitialTotalCapacity in class BaseRepeatedValueViewVector
      Parameters:
      numRecords - value count
      totalNumberOfElements - the total number of elements to allow for in this vector across all records.

    • getChildrenFromFields

      public List<FieldVector> getChildrenFromFields()
      Description copied from interface: FieldVector
      The returned list is the same size as the list passed to initializeChildrenFromFields.
      Specified by:
      getChildrenFromFields in interface FieldVector
      Returns:
      the children according to schema (empty for primitive types)

    • loadFieldBuffers

      public void loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers)
      Load the buffers associated with this Field.
      Specified by:
      loadFieldBuffers in interface FieldVector
      Parameters:
      fieldNode - the fieldNode
      ownBuffers - the buffers for this Field (own buffers only, children not included)

    • getFieldBuffers

      public List<ArrowBuf> getFieldBuffers()
      Description copied from interface: FieldVector
      Get the buffers of the fields, (same size as getFieldVectors() since it is their content).
      Specified by:
      getFieldBuffers in interface FieldVector
      Returns:
      the buffers containing the data for this vector (ready for reading)

    • exportCDataBuffers

      public void exportCDataBuffers(List<ArrowBuf> buffers, ArrowBuf buffersPtr, long nullValue)
      Export the buffers of the fields for C Data Interface. This method traverses the buffers and export buffer and buffer's memory address into a list of buffers and a pointer to the list of buffers.
      Specified by:
      exportCDataBuffers in interface FieldVector

    • allocateNew

      public void allocateNew() throws OutOfMemoryException
      Description copied from interface: ValueVector
      Allocate new buffers. ValueVector implements logic to determine how much to allocate.
      Specified by:
      allocateNew in interface ValueVector
      Throws:
      OutOfMemoryException - Thrown if no memory can be allocated.

    • allocateNewSafe

      public boolean allocateNewSafe()
      Description copied from interface: ValueVector
      Allocates new buffers. ValueVector implements logic to determine how much to allocate.
      Specified by:
      allocateNewSafe in interface ValueVector
      Overrides:
      allocateNewSafe in class BaseRepeatedValueViewVector
      Returns:
      Returns true if allocation was successful.

    • allocateValidityBuffer

      protected void allocateValidityBuffer(long size)

    • reAlloc

      public void reAlloc()
      Description copied from interface: ValueVector
      Allocate new buffer with double capacity, and copy data into the new buffer. Replace vector's buffer with new buffer, and release old one
      Specified by:
      reAlloc in interface ValueVector
      Overrides:
      reAlloc in class BaseRepeatedValueViewVector

    • reallocValidityAndSizeAndOffsetBuffers

      protected void reallocValidityAndSizeAndOffsetBuffers()

    • copyFromSafe

      public void copyFromSafe(int inIndex, int outIndex, ValueVector from)
      Description copied from interface: ValueVector
      Same as ValueVector.copyFrom(int, int, ValueVector) except that it handles the case when the capacity of the vector needs to be expanded before copy.
      Specified by:
      copyFromSafe in interface ValueVector
      Overrides:
      copyFromSafe in class BaseValueVector
      Parameters:
      inIndex - position to copy from in source vector
      outIndex - position to copy to in this vector
      from - source vector

    • copyFrom

      public void copyFrom(int inIndex, int outIndex, ValueVector from)
      Description copied from interface: ValueVector
      Copy a cell value from a particular index in source vector to a particular position in this vector.
      Specified by:
      copyFrom in interface ValueVector
      Overrides:
      copyFrom in class BaseValueVector
      Parameters:
      inIndex - position to copy from in source vector
      outIndex - position to copy to in this vector
      from - source vector

    • getDataVector

      public FieldVector getDataVector()
      Description copied from interface: RepeatedValueVector
      Get the data vector.
      Specified by:
      getDataVector in interface RepeatedValueVector
      Overrides:
      getDataVector in class BaseRepeatedValueViewVector
      Returns:
      the underlying data vector or null if none exists.

    • getTransferPair

      public TransferPair getTransferPair(String ref, BufferAllocator allocator)
      Description copied from interface: ValueVector
      To transfer quota responsibility.
      Specified by:
      getTransferPair in interface ValueVector
      Parameters:
      ref - the name of the vector
      allocator - the target allocator
      Returns:
      a transfer pair, creating a new target vector of the same type.

    • getTransferPair

      public TransferPair getTransferPair(Field field, BufferAllocator allocator)
      Description copied from interface: ValueVector
      To transfer quota responsibility.
      Specified by:
      getTransferPair in interface ValueVector
      Parameters:
      field - the Field object used by the target vector
      allocator - the target allocator
      Returns:
      a transfer pair, creating a new target vector of the same type.

    • getTransferPair

      public TransferPair getTransferPair(String ref, BufferAllocator allocator, CallBack callBack)
      Description copied from interface: ValueVector
      To transfer quota responsibility.
      Specified by:
      getTransferPair in interface ValueVector
      Parameters:
      ref - the name of the vector
      allocator - the target allocator
      callBack - A schema change callback.
      Returns:
      a transfer pair, creating a new target vector of the same type.

    • getTransferPair

      public TransferPair getTransferPair(Field field, BufferAllocator allocator, CallBack callBack)
      Description copied from interface: ValueVector
      To transfer quota responsibility.
      Specified by:
      getTransferPair in interface ValueVector
      Parameters:
      field - the Field object used by the target vector
      allocator - the target allocator
      callBack - A schema change callback.
      Returns:
      a transfer pair, creating a new target vector of the same type.

    • makeTransferPair

      public TransferPair makeTransferPair(ValueVector target)
      Description copied from interface: ValueVector
      Makes a new transfer pair used to transfer underlying buffers.
      Specified by:
      makeTransferPair in interface ValueVector
      Parameters:
      target - the target for the transfer
      Returns:
      a new transfer pair that is used to transfer underlying buffers into the target vector.

    • getValidityBufferAddress

      public long getValidityBufferAddress()
      Description copied from interface: FieldVector
      Gets the starting address of the underlying buffer associated with validity vector.
      Specified by:
      getValidityBufferAddress in interface FieldVector
      Returns:
      buffer address

    • getDataBufferAddress

      public long getDataBufferAddress()
      Description copied from interface: FieldVector
      Gets the starting address of the underlying buffer associated with data vector.
      Specified by:
      getDataBufferAddress in interface FieldVector
      Returns:
      buffer address

    • getOffsetBufferAddress

      public long getOffsetBufferAddress()
      Description copied from interface: FieldVector
      Gets the starting address of the underlying buffer associated with offset vector.
      Specified by:
      getOffsetBufferAddress in interface FieldVector
      Returns:
      buffer address

    • getValidityBuffer

      public ArrowBuf getValidityBuffer()
      Description copied from interface: ValueVector
      Gets the underlying buffer associated with validity vector.
      Specified by:
      getValidityBuffer in interface ValueVector
      Returns:
      buffer

    • getDataBuffer

      public ArrowBuf getDataBuffer()
      Description copied from interface: ValueVector
      Gets the underlying buffer associated with data vector.
      Specified by:
      getDataBuffer in interface ValueVector
      Returns:
      buffer

    • getOffsetBuffer

      public ArrowBuf getOffsetBuffer()
      Description copied from interface: ValueVector
      Gets the underlying buffer associated with offset vector.
      Specified by:
      getOffsetBuffer in interface ValueVector
      Returns:
      buffer

    • getSizeBuffer

      public ArrowBuf getSizeBuffer()

    • getSizeBufferAddress

      public long getSizeBufferAddress()

    • hashCode

      public int hashCode(int index)
      Get the hash code for the element at the given index.
      Specified by:
      hashCode in interface ValueVector
      Parameters:
      index - position of the element
      Returns:
      hash code for the element at the given index

    • hashCode

      public int hashCode(int index, ArrowBufHasher hasher)
      Get the hash code for the element at the given index.
      Specified by:
      hashCode in interface ValueVector
      Parameters:
      index - position of the element
      hasher - hasher to use
      Returns:
      hash code for the element at the given index

    • accept

      public <OUT, IN> OUT accept(VectorVisitor<OUT,IN> visitor, IN value)
      Description copied from interface: ValueVector
      Accept a generic VectorVisitor and return the result.
      Specified by:
      accept in interface ValueVector
      Type Parameters:
      OUT - the output result type.
      IN - the input data together with visitor.

    • getReaderImpl

      protected FieldReader getReaderImpl()
      Description copied from class: BaseValueVector
      Each vector has a different reader that implements the FieldReader interface. Overridden methods must make sure to return the correct concrete reader implementation.
      Specified by:
      getReaderImpl in class BaseValueVector
      Returns:
      Returns a lambda that initializes a reader when called.

    • getReader

      public UnionListReader getReader()
      Description copied from class: BaseValueVector
      Default implementation to create a reader for the vector. Depends on the individual vector class' implementation of BaseValueVector.getReaderImpl() to initialize the reader appropriately.
      Specified by:
      getReader in interface ValueVector
      Overrides:
      getReader in class BaseValueVector
      Returns:
      Concrete instance of FieldReader by using double-checked locking.

    • getBufferSize

      public int getBufferSize()
      Get the size (number of bytes) of underlying buffers used by this vector.
      Specified by:
      getBufferSize in interface ValueVector
      Overrides:
      getBufferSize in class BaseRepeatedValueViewVector
      Returns:
      size of underlying buffers.

    • getBufferSizeFor

      public int getBufferSizeFor(int valueCount)
      Get the size (number of bytes) of underlying buffers used by this.
      Specified by:
      getBufferSizeFor in interface ValueVector
      Overrides:
      getBufferSizeFor in class BaseRepeatedValueViewVector
      Parameters:
      valueCount - the number of values to assume this vector contains
      Returns:
      size of underlying buffers.

    • getField

      public Field getField()
      Get the field associated with the list view vector.
      Specified by:
      getField in interface ValueVector
      Returns:
      the field

    • getMinorType

      public Types.MinorType getMinorType()
      Get the minor type for the vector.
      Specified by:
      getMinorType in interface ValueVector
      Returns:
      the minor type

    • clear

      public void clear()
      Clear the vector data.
      Specified by:
      clear in interface ValueVector
      Overrides:
      clear in class BaseRepeatedValueViewVector

    • reset

      public void reset()
      Release the buffers associated with this vector.
      Specified by:
      reset in interface ValueVector
      Overrides:
      reset in class BaseRepeatedValueViewVector

    • getBuffers

      public ArrowBuf[] getBuffers(boolean clear)
      Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer, so it only should be used for in-context access. Also note that this buffer changes regularly, thus external classes shouldn't hold a reference to it (unless they change it).
      Specified by:
      getBuffers in interface ValueVector
      Overrides:
      getBuffers in class BaseRepeatedValueViewVector
      Parameters:
      clear - Whether to clear vector before returning, the buffers will still be refcounted but the returned array will be the only reference to them
      Returns:
      The underlying buffers that is used by this vector instance.

    • getObject

      public List<?> getObject(int index)
      Get the element in the list view vector at a particular index.
      Specified by:
      getObject in interface ValueVector
      Parameters:
      index - position of the element
      Returns:
      Object at given position

    • isNull

      public boolean isNull(int index)
      Check if an element at given index is null.
      Specified by:
      isNull in interface ValueVector
      Parameters:
      index - position of an element
      Returns:
      true if an element at given index is null, false otherwise

    • isEmpty

      public boolean isEmpty(int index)
      Check if an element at given index is an empty list.
      Specified by:
      isEmpty in class BaseRepeatedValueViewVector
      Parameters:
      index - position of an element
      Returns:
      true if an element at given index is an empty list or NULL, false otherwise

    • isSet

      public int isSet(int index)
      Same as isNull(int).
      Parameters:
      index - position of the element
      Returns:
      1 if element at given index is not null, 0 otherwise

    • getNullCount

      public int getNullCount()
      Get the number of elements that are null in the vector.
      Specified by:
      getNullCount in interface ValueVector
      Returns:
      the number of null elements.

    • getValueCapacity

      public int getValueCapacity()
      Get the value capacity by considering validity and offset capacity. Note that the size buffer capacity is not considered here since it has the same capacity as the offset buffer.
      Specified by:
      getValueCapacity in interface ValueVector
      Overrides:
      getValueCapacity in class BaseRepeatedValueViewVector
      Returns:
      the value capacity

    • setNull

      public void setNull(int index)
      Set the element at the given index to null.
      Specified by:
      setNull in interface FieldVector
      Parameters:
      index - the value to change

    • startNewValue

      public int startNewValue(int index)
      Start new value in the ListView vector.
      Overrides:
      startNewValue in class BaseRepeatedValueViewVector
      Parameters:
      index - index of the value to start
      Returns:
      offset of the new value

    • setOffset

      public void setOffset(int index, int value)
      Set the offset at the given index. Make sure to use this function after updating `field` vector and using `setValidity`
      Parameters:
      index - index of the value to set
      value - value to set

    • setSize

      public void setSize(int index, int value)
      Set the size at the given index. Make sure to use this function after using `setOffset`.
      Parameters:
      index - index of the value to set
      value - value to set

    • setValidity

      public void setValidity(int index, int value)
      Set the validity at the given index.
      Parameters:
      index - index of the value to set
      value - value to set (0 for unset and 1 for a set)

    • setValueCount

      public void setValueCount(int valueCount)
      Description copied from interface: ValueVector
      Set number of values in the vector.
      Specified by:
      setValueCount in interface ValueVector
      Overrides:
      setValueCount in class BaseRepeatedValueViewVector

    • getElementStartIndex

      public int getElementStartIndex(int index)
      Description copied from interface: BaseListVector
      Get data vector start index with the given list index.
      Specified by:
      getElementStartIndex in interface BaseListVector

    • getElementEndIndex

      public int getElementEndIndex(int index)
      Description copied from interface: BaseListVector
      Get data vector end index with the given list index.
      Specified by:
      getElementEndIndex in interface BaseListVector

    • addOrGetVector

      public <T extends ValueVector> AddOrGetResult<T> addOrGetVector(FieldType fieldType)
      Description copied from class: BaseRepeatedValueViewVector
      Initialize the data vector (and execute callback) if it hasn't already been done, returns the data vector.
      Specified by:
      addOrGetVector in interface PromotableVector
      Overrides:
      addOrGetVector in class BaseRepeatedValueViewVector

    • promoteToUnion

      public UnionVector promoteToUnion()
      Specified by:
      promoteToUnion in interface PromotableVector

    • getFieldInnerVectors

      @Deprecated public List<BufferBacked> getFieldInnerVectors()
      Deprecated.
      Description copied from interface: FieldVector
      Get the inner vectors.
      Specified by:
      getFieldInnerVectors in interface FieldVector
      Returns:
      the inner vectors for this field as defined by the TypeLayout

    • getWriter

      public UnionListViewWriter getWriter()

    • getValueCount

      public int getValueCount()
      Description copied from interface: ValueVector
      Gets the number of values.
      Specified by:
      getValueCount in interface ValueVector
      Overrides:
      getValueCount in class BaseRepeatedValueViewVector
      Returns:
      number of values in the vector

    • getDensity

      public double getDensity()
      Get the density of this ListVector.
      Returns:
      density

    • validate

      public void validate()
      Validating ListViewVector creation based on the specification guideline.
      Specified by:
      validate in interface ValueVector

    • endValue

      public void endValue(int index, int size)
      End the current value.
      Parameters:
      index - index of the value to end
      size - number of elements in the list that was written