org.apache.spark.sql

Class CarbonVectorProxy

java.lang.Object

org.apache.spark.sql.CarbonVectorProxy

public class CarbonVectorProxy
extends Object

Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API. This proxy class handles the complexity of spark 2.1 version related api changes since spark ColumnVector and ColumnarBatch interfaces are still evolving.

Constructor Summary

Constructors

Constructor and Description
CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode, int rowNum, org.apache.spark.sql.types.StructField[] structFileds) Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API.
CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode, org.apache.spark.sql.types.StructType outputSchema, int rowNum)

Method Summary

Modifier and Type	Method and Description
void	close() Called to close all the columns in this batch.
org.apache.spark.sql.execution.vectorized.ColumnVector	column(int ordinal) This API will return a columnvector from a batch of column vector rows based on the ordinal
org.apache.spark.sql.types.DataType	dataType(int ordinal)
Object	getColumnarBatch() Returns the row in this batch at `rowId`.
org.apache.spark.sql.execution.vectorized.ColumnVector	getColumnVector(int ordinal)
org.apache.spark.sql.catalyst.InternalRow	getRow(int rowId) Returns the row in this batch at `rowId`.
boolean	hasDictionary(int ordinal)
boolean	isNullAt(int rowId, int ordinal)
int	numRows() Returns the number of rows for read, including filtered rows.
void	putBoolean(int rowId, boolean value, int ordinal)
void	putByte(int rowId, byte value, int ordinal)
void	putByteArray(int rowId, byte[] value, int ordinal)
void	putByteArray(int rowId, byte[] value, int offset, int length, int ordinal)
void	putDecimal(int rowId, org.apache.spark.sql.types.Decimal value, int precision, int ordinal)
void	putDictionaryInt(int rowId, int value, int ordinal)
void	putDouble(int rowId, double value, int ordinal)
void	putDoubles(int rowId, int count, double value, int ordinal)
void	putFloat(int rowId, float value, int ordinal)
void	putInt(int rowId, int value, int ordinal)
void	putInts(int rowId, int count, int value, int ordinal)
void	putLong(int rowId, long value, int ordinal)
void	putLongs(int rowId, int count, long value, int ordinal)
void	putNotNull(int rowId, int ordinal)
void	putNotNulls(int rowId, int count, int ordinal)
void	putNull(int rowId, int ordinal)
void	putNulls(int rowId, int count, int ordinal)
void	putRowToColumnBatch(int rowId, Object value, int offset)
void	putShort(int rowId, short value, int ordinal)
void	putShorts(int rowId, int count, short value, int ordinal)
Object	reserveDictionaryIds(int capacity, int ordinal)
void	reset() Resets this column for writing.
void	resetDictionaryIds(int ordinal)
void	setDictionary(org.apache.carbondata.core.scan.result.vector.CarbonDictionary dictionary, int ordinal)
void	setNumRows(int numRows) Sets the number of rows in this batch.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CarbonVectorProxy

public CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode,
                         int rowNum,
                         org.apache.spark.sql.types.StructField[] structFileds)

Adapter class which handles the columnar vector reading of the carbondata based on the spark ColumnVector and ColumnarBatch API. This proxy class handles the complexity of spark 2.3 version related api changes since spark ColumnVector and ColumnarBatch interfaces are still evolving.

Parameters:

memMode - which represent the type onheap or offheap vector.

rowNum - rows number for vector reading

structFileds, - metadata related to current schema of table.

CarbonVectorProxy

public CarbonVectorProxy(org.apache.spark.memory.MemoryMode memMode,
                         org.apache.spark.sql.types.StructType outputSchema,
                         int rowNum)

Method Detail

getColumnVector

public org.apache.spark.sql.execution.vectorized.ColumnVector getColumnVector(int ordinal)

setNumRows

public void setNumRows(int numRows)

Sets the number of rows in this batch.

reserveDictionaryIds

public Object reserveDictionaryIds(int capacity,
                                   int ordinal)

numRows

public int numRows()

Returns the number of rows for read, including filtered rows.

setDictionary

public void setDictionary(org.apache.carbondata.core.scan.result.vector.CarbonDictionary dictionary,
                          int ordinal)

putNull

public void putNull(int rowId,
                    int ordinal)

putNulls

public void putNulls(int rowId,
                     int count,
                     int ordinal)

close

public void close()

Called to close all the columns in this batch. It is not valid to access the data after calling this. This must be called at the end to clean up memory allocations.

getRow

public org.apache.spark.sql.catalyst.InternalRow getRow(int rowId)

Returns the row in this batch at `rowId`. Returned row is reused across calls.

getColumnarBatch

public Object getColumnarBatch()

Returns the row in this batch at `rowId`. Returned row is reused across calls.

resetDictionaryIds

public void resetDictionaryIds(int ordinal)

column

public org.apache.spark.sql.execution.vectorized.ColumnVector column(int ordinal)

This API will return a columnvector from a batch of column vector rows based on the ordinal

Parameters:

ordinal -

Returns:

hasDictionary

public boolean hasDictionary(int ordinal)

reset

public void reset()

Resets this column for writing. The currently stored values are no longer accessible.

putRowToColumnBatch

public void putRowToColumnBatch(int rowId,
                                Object value,
                                int offset)

putBoolean

public void putBoolean(int rowId,
                       boolean value,
                       int ordinal)

putByte

public void putByte(int rowId,
                    byte value,
                    int ordinal)

putShort

public void putShort(int rowId,
                     short value,
                     int ordinal)

putInt

public void putInt(int rowId,
                   int value,
                   int ordinal)

putFloat

public void putFloat(int rowId,
                     float value,
                     int ordinal)

putLong

public void putLong(int rowId,
                    long value,
                    int ordinal)

putDouble

public void putDouble(int rowId,
                      double value,
                      int ordinal)

putByteArray

public void putByteArray(int rowId,
                         byte[] value,
                         int ordinal)

putInts

public void putInts(int rowId,
                    int count,
                    int value,
                    int ordinal)

putShorts

public void putShorts(int rowId,
                      int count,
                      short value,
                      int ordinal)

putLongs

public void putLongs(int rowId,
                     int count,
                     long value,
                     int ordinal)

putDecimal

public void putDecimal(int rowId,
                       org.apache.spark.sql.types.Decimal value,
                       int precision,
                       int ordinal)

putDoubles

public void putDoubles(int rowId,
                       int count,
                       double value,
                       int ordinal)

putByteArray

public void putByteArray(int rowId,
                         byte[] value,
                         int offset,
                         int length,
                         int ordinal)

isNullAt

public boolean isNullAt(int rowId,
                        int ordinal)

dataType

public org.apache.spark.sql.types.DataType dataType(int ordinal)

putNotNull

public void putNotNull(int rowId,
                       int ordinal)

putNotNulls

public void putNotNulls(int rowId,
                        int count,
                        int ordinal)

putDictionaryInt

public void putDictionaryInt(int rowId,
                             int value,
                             int ordinal)