org.apache.arrow.vector.complex

Class LargeListVector

java.lang.Object

org.apache.arrow.vector.BaseValueVector

org.apache.arrow.vector.complex.LargeListVector

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ValueVector>, PromotableVector, RepeatedValueVector, DensityAwareVector, FieldVector, ValueVector

public class LargeListVector
extends BaseValueVector
implements RepeatedValueVector, FieldVector, PromotableVector

A list vector contains lists of a specific type of elements. Its structure contains 3 elements.

1. A validity buffer.
2. An offset buffer, that denotes lists boundaries.
3. A child data vector that contains the elements of lists.

This is the LargeList variant of list, it has a 64-bit wide offset

WARNING: Currently Arrow in Java doesn't support 64-bit vectors. This class follows the expected behaviour of a LargeList but doesn't actually support allocating a 64-bit vector. It has little use until 64-bit vectors are supported and should be used with caution. todo review checkedCastToInt usage in this class. Once int64 indexed vectors are supported these checks aren't needed.

Field Summary

Fields

Modifier and Type	Field and Description
protected CallBack	callBack
static String	DATA_VECTOR_NAME
static FieldVector	DEFAULT_DATA_VECTOR
protected String	defaultDataVectorName
static byte	OFFSET_WIDTH
protected long	offsetAllocationSizeInBytes
protected ArrowBuf	offsetBuffer
protected UnionLargeListReader	reader
protected ArrowBuf	validityBuffer
protected int	valueCount
protected FieldVector	vector

Fields inherited from class org.apache.arrow.vector.BaseValueVector

allocator, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY

Fields inherited from interface org.apache.arrow.vector.complex.RepeatedValueVector

DEFAULT_REPEAT_PER_RECORD

Constructor Summary

Constructors

Constructor and Description
LargeListVector(String name, BufferAllocator allocator, FieldType fieldType, CallBack callBack) Constructs a new instance.

Method Summary

Modifier and Type	Method and Description
<OUT,IN> OUT	accept(VectorVisitor<OUT,IN> visitor, IN value) Accept a generic VectorVisitor and return the result.
<T extends ValueVector> AddOrGetResult<T>	addOrGetVector(FieldType fieldType) Initialize the data vector (and execute callback) if it hasn't already been done, returns the data vector.
void	allocateNew() Same as allocateNewSafe().
boolean	allocateNewSafe() Allocate memory for the vector.
protected void	allocateOffsetBuffer(long size)
void	clear() Release any owned ArrowBuf and reset the ValueVector to the initial state.
void	copyFrom(int inIndex, int outIndex, ValueVector from) Copy a cell value from a particular index in source vector to a particular position in this vector.
void	copyFromSafe(int inIndex, int outIndex, ValueVector from) Same as copyFrom(int, int, ValueVector) except that it handles the case when the capacity of the vector needs to be expanded before copy.
static LargeListVector	empty(String name, BufferAllocator allocator)
void	endValue(int index, long size) End the current value.
ArrowBuf[]	getBuffers(boolean clear) Return the underlying buffers associated with this vector.
int	getBufferSize() Get the size (number of bytes) of underlying buffers used by this vector.
int	getBufferSizeFor(int valueCount) Returns the number of bytes that is used by this vector if it holds the given number of values.
List<FieldVector>	getChildrenFromFields() The returned list is the same size as the list passed to initializeChildrenFromFields.
ArrowBuf	getDataBuffer() Gets the underlying buffer associated with data vector.
long	getDataBufferAddress() Gets the starting address of the underlying buffer associated with data vector.
FieldVector	getDataVector() Get the inner data vector for this list vector.
double	getDensity() Get the density of this ListVector.
long	getElementEndIndex(int index)
long	getElementStartIndex(int index)
Field	getField() Get information about how this field is materialized.
List<ArrowBuf>	getFieldBuffers() Get the buffers belonging to this vector.
List<BufferBacked>	getFieldInnerVectors() Deprecated.
int	getLastSet()
Types.MinorType	getMinorType()
String	getName() Gets the name of the vector.
int	getNullCount() Get the number of elements that are null in the vector.
List<?>	getObject(int index) Get the element in the list vector at a particular index.
ArrowBuf	getOffsetBuffer() Gets the underlying buffer associated with offset vector.
long	getOffsetBufferAddress() Gets the starting address of the underlying buffer associated with offset vector.
protected int	getOffsetBufferValueCapacity()
UInt4Vector	getOffsetVector() Get the offset vector.
UnionLargeListReader	getReader() Get a reader for this vector.
TransferPair	getTransferPair(String ref, BufferAllocator allocator)
TransferPair	getTransferPair(String ref, BufferAllocator allocator, CallBack callBack)
ArrowBuf	getValidityBuffer() Gets the underlying buffer associated with validity vector.
long	getValidityBufferAddress() Gets the starting address of the underlying buffer associated with validity vector.
int	getValueCapacity() Get the current value capacity for the vector.
int	getValueCount() Gets the number of values.
UnionLargeListWriter	getWriter()
int	hashCode(int index) Returns hashCode of element in index with the default hasher.
int	hashCode(int index, ArrowBufHasher hasher) Returns hashCode of element in index with the given hasher.
void	initializeChildrenFromFields(List<Field> children) Initializes the child vectors to be later loaded with loadBuffers.
protected void	invalidateReader()
boolean	isEmpty(int index) Check if element at given index is empty list.
boolean	isNull(int index) Check if element at given index is null.
int	isSet(int index) Same as isNull(int).
void	loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) Load the buffers of this vector with provided source buffers.
TransferPair	makeTransferPair(ValueVector target) Makes a new transfer pair used to transfer underlying buffers.
UnionVector	promoteToUnion()
void	reAlloc() Resize the vector to increase the capacity.
protected void	reallocOffsetBuffer()
protected void	replaceDataVector(FieldVector v)
void	reset() Reset the ValueVector to the initial state without releasing any owned ArrowBuf.
void	setInitialCapacity(int numRecords) Set the initial record capacity.
void	setInitialCapacity(int numRecords, double density) Specialized version of setInitialCapacity() for ListVector.
void	setLastSet(int value)
void	setNotNull(int index) Sets the list at index to be not-null.
void	setNull(int index) Sets list at index to be null.
void	setValueCount(int valueCount) Sets the value count for the vector.
long	startNewValue(long index) Start a new value in the list vector.

Methods inherited from class org.apache.arrow.vector.BaseValueVector

checkBufRefs, close, getAllocator, getTransferPair, getValidityBufferSizeFromCount, iterator, releaseBuffer, toString, transferBuffer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.arrow.vector.ValueVector

close, getAllocator, getTransferPair

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Field Detail

DEFAULT_DATA_VECTOR

public static final FieldVector DEFAULT_DATA_VECTOR

DATA_VECTOR_NAME

public static final String DATA_VECTOR_NAME

See Also:

Constant Field Values

OFFSET_WIDTH

public static final byte OFFSET_WIDTH

See Also:

Constant Field Values

offsetBuffer

protected ArrowBuf offsetBuffer

vector

protected FieldVector vector

callBack

protected final CallBack callBack

valueCount

protected int valueCount

offsetAllocationSizeInBytes

protected long offsetAllocationSizeInBytes

defaultDataVectorName

protected String defaultDataVectorName

validityBuffer

protected ArrowBuf validityBuffer

reader

protected UnionLargeListReader reader

Constructor Detail

LargeListVector

public LargeListVector(String name,
                       BufferAllocator allocator,
                       FieldType fieldType,
                       CallBack callBack)

Constructs a new instance.

Parameters:

name - The name of the instance.

allocator - The allocator to use for allocating/reallocating buffers.

fieldType - The type of this list.

callBack - A schema change callback.

Method Detail

empty

public static LargeListVector empty(String name,
                                    BufferAllocator allocator)

initializeChildrenFromFields

public void initializeChildrenFromFields(List<Field> children)

Description copied from interface: FieldVector

Initializes the child vectors to be later loaded with loadBuffers.

Specified by:

initializeChildrenFromFields in interface FieldVector

Parameters:

children - the schema

setInitialCapacity

public void setInitialCapacity(int numRecords)

Description copied from interface: ValueVector

Set the initial record capacity.

Specified by:

setInitialCapacity in interface ValueVector

Parameters:

numRecords - the initial record capacity.

setInitialCapacity

public void setInitialCapacity(int numRecords,
                               double density)

Specialized version of setInitialCapacity() for ListVector. This is used by some callers when they want to explicitly control and be conservative about memory allocated for inner data vector. This is very useful when we are working with memory constraints for a query and have a fixed amount of memory reserved for the record batch. In such cases, we are likely to face OOM or related problems when we reserve memory for a record batch with value count x and do setInitialCapacity(x) such that each vector allocates only what is necessary and not the default amount but the multiplier forces the memory requirement to go beyond what was needed.

Specified by:

setInitialCapacity in interface DensityAwareVector

Parameters:

numRecords - value count

density - density of ListVector. Density is the average size of list per position in the List vector. For example, a density value of 10 implies each position in the list vector has a list of 10 values. A density value of 0.1 implies out of 10 positions in the list vector, 1 position has a list of size 1 and remaining positions are null (no lists) or empty lists. This helps in tightly controlling the memory we provision for inner data vector.

getDensity

public double getDensity()

Get the density of this ListVector.

Returns:

density

getChildrenFromFields

public List<FieldVector> getChildrenFromFields()

Description copied from interface: FieldVector

The returned list is the same size as the list passed to initializeChildrenFromFields.

Specified by:

getChildrenFromFields in interface FieldVector

Returns:

the children according to schema (empty for primitive types)

loadFieldBuffers

public void loadFieldBuffers(ArrowFieldNode fieldNode,
                             List<ArrowBuf> ownBuffers)

Load the buffers of this vector with provided source buffers. The caller manages the source buffers and populates them before invoking this method.

Specified by:

loadFieldBuffers in interface FieldVector

Parameters:

fieldNode - the fieldNode indicating the value count

ownBuffers - the buffers for this Field (own buffers only, children not included)

getFieldBuffers

public List<ArrowBuf> getFieldBuffers()

Get the buffers belonging to this vector.

Specified by:

getFieldBuffers in interface FieldVector

Returns:

the inner buffers.

getFieldInnerVectors

@Deprecated
public List<BufferBacked> getFieldInnerVectors()

Deprecated.

Description copied from interface: FieldVector

Get the inner vectors.

Specified by:

getFieldInnerVectors in interface FieldVector

Returns:

the inner vectors for this field as defined by the TypeLayout

allocateNew

public void allocateNew()
                 throws OutOfMemoryException

Same as allocateNewSafe().

Specified by:

allocateNew in interface ValueVector

Throws:

OutOfMemoryException - Thrown if no memory can be allocated.

allocateNewSafe

public boolean allocateNewSafe()

Allocate memory for the vector. We internally use a default value count of 4096 to allocate memory for at least these many elements in the vector.

Specified by:

allocateNewSafe in interface ValueVector

Returns:

false if memory allocation fails, true otherwise.

allocateOffsetBuffer

protected void allocateOffsetBuffer(long size)

reAlloc

public void reAlloc()

Resize the vector to increase the capacity. The internal behavior is to double the current value capacity.

Specified by:

reAlloc in interface ValueVector

reallocOffsetBuffer

protected void reallocOffsetBuffer()

copyFromSafe

public void copyFromSafe(int inIndex,
                         int outIndex,
                         ValueVector from)

Same as copyFrom(int, int, ValueVector) except that it handles the case when the capacity of the vector needs to be expanded before copy.

Specified by:

copyFromSafe in interface ValueVector

Overrides:

copyFromSafe in class BaseValueVector

Parameters:

inIndex - position to copy from in source vector

outIndex - position to copy to in this vector

from - source vector

copyFrom

public void copyFrom(int inIndex,
                     int outIndex,
                     ValueVector from)

Copy a cell value from a particular index in source vector to a particular position in this vector.

Specified by:

copyFrom in interface ValueVector

Overrides:

copyFrom in class BaseValueVector

Parameters:

inIndex - position to copy from in source vector

outIndex - position to copy to in this vector

from - source vector

getOffsetVector

public UInt4Vector getOffsetVector()

Description copied from interface: RepeatedValueVector

Get the offset vector.

Specified by:

getOffsetVector in interface RepeatedValueVector

Returns:

the underlying offset vector or null if none exists.

getDataVector

public FieldVector getDataVector()

Get the inner data vector for this list vector.

Specified by:

getDataVector in interface RepeatedValueVector

Returns:

data vector

getTransferPair

public TransferPair getTransferPair(String ref,
                                    BufferAllocator allocator)

Specified by:

getTransferPair in interface ValueVector

getTransferPair

public TransferPair getTransferPair(String ref,
                                    BufferAllocator allocator,
                                    CallBack callBack)

Specified by:

getTransferPair in interface ValueVector

makeTransferPair

public TransferPair makeTransferPair(ValueVector target)

Description copied from interface: ValueVector

Makes a new transfer pair used to transfer underlying buffers.

Specified by:

makeTransferPair in interface ValueVector

Parameters:

target - the target for the transfer

Returns:

a new transfer pair that is used to transfer underlying buffers into the target vector.

getValidityBufferAddress

public long getValidityBufferAddress()

Description copied from interface: FieldVector

Gets the starting address of the underlying buffer associated with validity vector.

Specified by:

getValidityBufferAddress in interface FieldVector

Returns:

buffer address

getDataBufferAddress

public long getDataBufferAddress()

Description copied from interface: FieldVector

Gets the starting address of the underlying buffer associated with data vector.

Specified by:

getDataBufferAddress in interface FieldVector

Returns:

buffer address

getOffsetBufferAddress

public long getOffsetBufferAddress()

Description copied from interface: FieldVector

Gets the starting address of the underlying buffer associated with offset vector.

Specified by:

getOffsetBufferAddress in interface FieldVector

Returns:

buffer address

getValidityBuffer

public ArrowBuf getValidityBuffer()

Description copied from interface: ValueVector

Gets the underlying buffer associated with validity vector.

Specified by:

getValidityBuffer in interface ValueVector

Returns:

buffer

getDataBuffer

public ArrowBuf getDataBuffer()

Description copied from interface: ValueVector

Gets the underlying buffer associated with data vector.

Specified by:

getDataBuffer in interface ValueVector

Returns:

buffer

getOffsetBuffer

public ArrowBuf getOffsetBuffer()

Description copied from interface: ValueVector

Gets the underlying buffer associated with offset vector.

Specified by:

getOffsetBuffer in interface ValueVector

Returns:

buffer

getValueCount

public int getValueCount()

Description copied from interface: ValueVector

Gets the number of values.

Specified by:

getValueCount in interface ValueVector

Returns:

number of values in the vector

hashCode

public int hashCode(int index)

Description copied from interface: ValueVector

Returns hashCode of element in index with the default hasher.

Specified by:

hashCode in interface ValueVector

hashCode

public int hashCode(int index,
                    ArrowBufHasher hasher)

Description copied from interface: ValueVector

Returns hashCode of element in index with the given hasher.

Specified by:

hashCode in interface ValueVector

accept

public <OUT,IN> OUT accept(VectorVisitor<OUT,IN> visitor,
                           IN value)

Description copied from interface: ValueVector

Accept a generic VectorVisitor and return the result.

Specified by:

accept in interface ValueVector

Type Parameters:

OUT - the output result type.

IN - the input data together with visitor.

getWriter

public UnionLargeListWriter getWriter()

replaceDataVector

protected void replaceDataVector(FieldVector v)

promoteToUnion

public UnionVector promoteToUnion()

Specified by:

promoteToUnion in interface PromotableVector

getReader

public UnionLargeListReader getReader()

Description copied from interface: ValueVector

Get a reader for this vector.

Specified by:

getReader in interface ValueVector

Returns:

a field reader that supports reading values from this vector.

addOrGetVector

public <T extends ValueVector> AddOrGetResult<T> addOrGetVector(FieldType fieldType)

Initialize the data vector (and execute callback) if it hasn't already been done, returns the data vector.

Specified by:

addOrGetVector in interface PromotableVector

getBufferSize

public int getBufferSize()

Get the size (number of bytes) of underlying buffers used by this vector.

Specified by:

getBufferSize in interface ValueVector

Returns:

size of underlying buffers.

getBufferSizeFor

public int getBufferSizeFor(int valueCount)

Description copied from interface: ValueVector

Returns the number of bytes that is used by this vector if it holds the given number of values. The result will be the same as if setValueCount() were called, followed by calling getBufferSize(), but without any of the closing side-effects that setValueCount() implies wrt finishing off the population of a vector. Some operations might wish to use this to determine how much memory has been used by a vector so far, even though it is not finished being populated.

Specified by:

getBufferSizeFor in interface ValueVector

Parameters:

valueCount - the number of values to assume this vector contains

Returns:

the buffer size if this vector is holding valueCount values

getField

public Field getField()

Description copied from interface: ValueVector

Get information about how this field is materialized.

Specified by:

getField in interface ValueVector

Returns:

the field corresponding to this vector

getMinorType

public Types.MinorType getMinorType()

Specified by:

getMinorType in interface ValueVector

getName

public String getName()

Description copied from interface: ValueVector

Gets the name of the vector.

Specified by:

getName in interface ValueVector

Specified by:

getName in class BaseValueVector

Returns:

the name of the vector.

clear

public void clear()

Description copied from interface: ValueVector

Release any owned ArrowBuf and reset the ValueVector to the initial state. If the vector has any child vectors, they will also be cleared.

Specified by:

clear in interface ValueVector

Overrides:

clear in class BaseValueVector

reset

public void reset()

Description copied from interface: ValueVector

Reset the ValueVector to the initial state without releasing any owned ArrowBuf. Buffer capacities will remain unchanged and any previous data will be zeroed out. This includes buffers for data, validity, offset, etc. If the vector has any child vectors, they will also be reset.

Specified by:

reset in interface ValueVector

getBuffers

public ArrowBuf[] getBuffers(boolean clear)

Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer so it only should be used for in-context access. Also note that this buffer changes regularly thus external classes shouldn't hold a reference to it (unless they change it).

Specified by:

getBuffers in interface ValueVector

Parameters:

clear - Whether to clear vector before returning; the buffers will still be refcounted but the returned array will be the only reference to them

Returns:

The underlying buffers that is used by this vector instance.

invalidateReader

protected void invalidateReader()

getObject

public List<?> getObject(int index)

Get the element in the list vector at a particular index.

Specified by:

getObject in interface ValueVector

Parameters:

index - position of the element

Returns:

Object at given position

isNull

public boolean isNull(int index)

Check if element at given index is null.

Specified by:

isNull in interface ValueVector

Parameters:

index - position of element

Returns:

true if element at given index is null, false otherwise

isEmpty

public boolean isEmpty(int index)

Check if element at given index is empty list.

Parameters:

index - position of element

Returns:

true if element at given index is empty list or NULL, false otherwise

isSet

public int isSet(int index)

Same as isNull(int).

Parameters:

index - position of element

Returns:

1 if element at given index is not null, 0 otherwise

getNullCount

public int getNullCount()

Get the number of elements that are null in the vector.

Specified by:

getNullCount in interface ValueVector

Returns:

the number of null elements.

getValueCapacity

public int getValueCapacity()

Get the current value capacity for the vector.

Specified by:

getValueCapacity in interface ValueVector

Returns:

number of elements that vector can hold.

getOffsetBufferValueCapacity

protected int getOffsetBufferValueCapacity()

setNotNull

public void setNotNull(int index)

Sets the list at index to be not-null. Reallocates validity buffer if index is larger than current capacity.

setNull

public void setNull(int index)

Sets list at index to be null.

Parameters:

index - position in vector

startNewValue

public long startNewValue(long index)

Start a new value in the list vector.

Parameters:

index - index of the value to start

endValue

public void endValue(int index,
                     long size)

End the current value.

Parameters:

index - index of the value to end

size - number of elements in the list that was written

setValueCount

public void setValueCount(int valueCount)

Sets the value count for the vector.

Important note: The underlying vector does not support 64-bit allocations yet. This may throw if attempting to hold larger than what a 32-bit vector can store.

Specified by:

setValueCount in interface ValueVector

Parameters:

valueCount - value count

setLastSet

public void setLastSet(int value)

getLastSet

public int getLastSet()

getElementStartIndex

public long getElementStartIndex(int index)

getElementEndIndex

public long getElementEndIndex(int index)