org.apache.arrow.vector

Class BaseVariableWidthVector

java.lang.Object

org.apache.arrow.vector.BaseValueVector

org.apache.arrow.vector.BaseVariableWidthVector

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<ValueVector>, FieldVector, ValueVector, VariableWidthVector, VectorDefinitionSetter

Direct Known Subclasses:

VarBinaryVector, VarCharVector

public abstract class BaseVariableWidthVector
extends BaseValueVector
implements VariableWidthVector, FieldVector, VectorDefinitionSetter

Field Summary

Fields

Modifier and Type	Field and Description
protected static byte[]	emptyByteArray
protected Field	field
protected int	lastSet
static int	OFFSET_WIDTH
protected ArrowBuf	offsetBuffer
protected ArrowBuf	validityBuffer
protected ArrowBuf	valueBuffer
protected int	valueCount

Fields inherited from class org.apache.arrow.vector.BaseValueVector

allocator, INITIAL_VALUE_ALLOCATION, MAX_ALLOCATION_SIZE, MAX_ALLOCATION_SIZE_PROPERTY, name

Constructor Summary

Constructors

Constructor and Description
BaseVariableWidthVector(String name, BufferAllocator allocator, FieldType fieldType)

Method Summary

Methods

Modifier and Type	Method and Description
void	allocateNew() Same as allocateNewSafe().
void	allocateNew(int totalBytes, int valueCount) Allocate memory for the vector to support storing at least the provided number of elements in the vector.
boolean	allocateNewSafe() Allocate memory for the vector.
void	clear() Same as close()
void	close() Close the vector and release the associated buffers.
void	fillEmpties(int index) Create holes in the vector upto the given index (exclusive).
protected void	fillHoles(int index) * helper methods for setters * *
static byte[]	get(ArrowBuf data, ArrowBuf offset, int index) Method used by Json Writer to read a variable width element from the variable width vector and write to Json.
ArrowBuf[]	getBuffers(boolean clear) Return the underlying buffers associated with this vector.
int	getBufferSize() Get the size (number of bytes) of underlying buffers used by this vector
int	getBufferSizeFor(int valueCount) Get the potential buffer size for a particular number of records.
int	getByteCapacity() Get the size (number of bytes) of underlying data buffer.
List<FieldVector>	getChildrenFromFields() Get the inner child vectors.
int	getCurrentSizeInBytes()
ArrowBuf	getDataBuffer() Get the buffer that stores the data for elements in the vector.
long	getDataBufferAddress() Get the memory address of buffer that stores the data for elements in the vector.
Field	getField() Get information about how this field is materialized.
List<ArrowBuf>	getFieldBuffers() Get the buffers belonging to this vector
List<BufferBacked>	getFieldInnerVectors() Deprecated.
int	getLastSet() Get the index of last non-null element in the vector.
int	getNullCount() Get the number of elements that are null in the vector
ArrowBuf	getOffsetBuffer() buffer that stores the offsets for elements in the vector.
long	getOffsetBufferAddress() Get the memory address of buffer that stores the offsets for elements in the vector.
long	getStartEnd(int index) Get the starting position (offset) in the data stream for a given element in the vector.
protected int	getstartOffset(int index)
TransferPair	getTransferPair(BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
abstract TransferPair	getTransferPair(String ref, BufferAllocator allocator) Construct a transfer pair of this vector and another vector of same type.
TransferPair	getTransferPair(String ref, BufferAllocator allocator, CallBack callBack) Construct a transfer pair of this vector and another vector of same type.
ArrowBuf	getValidityBuffer() Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
long	getValidityBufferAddress() Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
int	getValueCapacity() Get the current value capacity for the vector
int	getValueCount() Get the value count of vector.
int	getValueLength(int index) Get the variable length element at specified index as Text.
protected void	handleSafe(int index, int dataLength)
void	initializeChildrenFromFields(List<Field> children) Initialize the children in schema for this Field.
boolean	isNull(int index) Check if element at given index is null.
boolean	isSafe(int index) Check if the given index is within the current value capacity of the vector
int	isSet(int index) Same as isNull(int).
void	loadFieldBuffers(ArrowFieldNode fieldNode, List<ArrowBuf> ownBuffers) Load the buffers of this vector with provided source buffers.
void	reAlloc() Resize the vector to increase the capacity.
void	reallocDataBuffer() Reallocate the data buffer.
void	reallocValidityAndOffsetBuffers() Reallocate the validity and offset buffers for this vector.
void	reset() Reset the vector to initial state.
static ArrowBuf	set(ArrowBuf buffer, BufferAllocator allocator, int valueCount, int index, int value) Method used by Json Reader to explicitly set the offsets of the variable width vector data.
void	set(int index, byte[] value) Set the variable length element at the specified index to the supplied byte array.
void	set(int index, byte[] value, int start, int length) Set the variable length element at the specified index to the supplied byte array.
void	set(int index, ByteBuffer value, int start, int length) Set the variable length element at the specified index to the content in supplied ByteBuffer
void	set(int index, int start, int length, ArrowBuf buffer) Store the given value at a particular position in the vector.
void	set(int index, int isSet, int start, int end, ArrowBuf buffer) Store the given value at a particular position in the vector.
protected void	setBytes(int index, byte[] value, int start, int length)
void	setIndexDefined(int index) Mark the particular position in the vector as non-null.
void	setInitialCapacity(int valueCount) Sets the desired value capacity for the vector.
void	setLastSet(int value) Set the index of last non-null element in the vector.
void	setNull(int index) Set the element at the given index to null.
void	setSafe(int index, byte[] value) Same as set(int, byte[]) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.
void	setSafe(int index, byte[] value, int start, int length) Same as set(int, byte[], int, int) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.
void	setSafe(int index, ByteBuffer value, int start, int length) Same as set(int, ByteBuffer, int, int) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.
void	setSafe(int index, int start, int length, ArrowBuf buffer) Same as set(int, int, int, int, ArrowBuf) except that it handles the case when index is greater than or equal to current value capacity of the vector.
void	setSafe(int index, int isSet, int start, int end, ArrowBuf buffer) Same as set(int, int, int, int, ArrowBuf) except that it handles the case when index is greater than or equal to current value capacity of the vector.
void	setValueCount(int valueCount) Sets the value count for the vector
void	setValueLengthSafe(int index, int length) Sets the value length for an element.
void	splitAndTransferTo(int startIndex, int length, BaseVariableWidthVector target) Slice this vector at desired index and length and transfer the corresponding data to the target vector.
void	transferTo(BaseVariableWidthVector target) Transfer this vector'data to another vector.
void	zeroVector() zero out the vector and the data in associated buffers.

Methods inherited from class org.apache.arrow.vector.BaseValueVector

checkBufRefs, compareTypes, getAllocator, getValidityBufferSizeFromCount, iterator, releaseBuffer, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.arrow.vector.ValueVector

getAllocator, getMinorType, getObject, getReader, makeTransferPair

Methods inherited from interface java.lang.Iterable

iterator

Field Detail

OFFSET_WIDTH

public static final int OFFSET_WIDTH

See Also:

Constant Field Values

emptyByteArray

protected static final byte[] emptyByteArray

validityBuffer

protected ArrowBuf validityBuffer

valueBuffer

protected ArrowBuf valueBuffer

offsetBuffer

protected ArrowBuf offsetBuffer

valueCount

protected int valueCount

lastSet

protected int lastSet

field

protected final Field field

Constructor Detail

BaseVariableWidthVector

public BaseVariableWidthVector(String name,
                       BufferAllocator allocator,
                       FieldType fieldType)

Method Detail

getValidityBuffer

public ArrowBuf getValidityBuffer()

Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector. Consider it as a buffer for internal bit vector data structure.

Specified by:

getValidityBuffer in interface ValueVector

Returns:

buffer

getDataBuffer

public ArrowBuf getDataBuffer()

Get the buffer that stores the data for elements in the vector.

Specified by:

getDataBuffer in interface ValueVector

Returns:

buffer

getOffsetBuffer

public ArrowBuf getOffsetBuffer()

buffer that stores the offsets for elements in the vector. This operation is not supported for fixed-width vectors.

Specified by:

getOffsetBuffer in interface ValueVector

Returns:

buffer

getOffsetBufferAddress

public long getOffsetBufferAddress()

Get the memory address of buffer that stores the offsets for elements in the vector.

Specified by:

getOffsetBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getValidityBufferAddress

public long getValidityBufferAddress()

Get the memory address of buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.

Specified by:

getValidityBufferAddress in interface FieldVector

Returns:

starting address of the buffer

getDataBufferAddress

public long getDataBufferAddress()

Get the memory address of buffer that stores the data for elements in the vector.

Specified by:

getDataBufferAddress in interface FieldVector

Returns:

starting address of the buffer

setInitialCapacity

public void setInitialCapacity(int valueCount)

Sets the desired value capacity for the vector. This function doesn't allocate any memory for the vector.

Specified by:

setInitialCapacity in interface ValueVector

Parameters:

valueCount - desired number of elements in the vector

getValueCapacity

public int getValueCapacity()

Get the current value capacity for the vector

Specified by:

getValueCapacity in interface ValueVector

Returns:

number of elements that vector can hold.

zeroVector

public void zeroVector()

zero out the vector and the data in associated buffers.

reset

public void reset()

Reset the vector to initial state. Same as zeroVector(). Note that this method doesn't release any memory.

close

public void close()

Close the vector and release the associated buffers.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ValueVector

Overrides:

close in class BaseValueVector

clear

public void clear()

Same as close()

Specified by:

clear in interface ValueVector

Overrides:

clear in class BaseValueVector

getFieldInnerVectors

@Deprecated
public List<BufferBacked> getFieldInnerVectors()

Deprecated.

Specified by:

getFieldInnerVectors in interface FieldVector

Returns:

the inner vectors for this field as defined by the TypeLayout

initializeChildrenFromFields

public void initializeChildrenFromFields(List<Field> children)

Initialize the children in schema for this Field. This operation is a NO-OP for scalar types since they don't have any children.

Specified by:

initializeChildrenFromFields in interface FieldVector

Parameters:

children - the schema

Throws:

IllegalArgumentException - if children is a non-empty list for scalar types.

getChildrenFromFields

public List<FieldVector> getChildrenFromFields()

Get the inner child vectors.

Specified by:

getChildrenFromFields in interface FieldVector

Returns:

list of child vectors for complex types, empty list for scalar vector types

loadFieldBuffers

public void loadFieldBuffers(ArrowFieldNode fieldNode,
                    List<ArrowBuf> ownBuffers)

Load the buffers of this vector with provided source buffers. The caller manages the source buffers and populates them before invoking this method.

Specified by:

loadFieldBuffers in interface FieldVector

Parameters:

fieldNode - the fieldNode indicating the value count

ownBuffers - the buffers for this Field (own buffers only, children not included)

getFieldBuffers

public List<ArrowBuf> getFieldBuffers()

Get the buffers belonging to this vector

Specified by:

getFieldBuffers in interface FieldVector

Returns:

the inner buffers.

allocateNew

public void allocateNew()

Same as allocateNewSafe().

Specified by:

allocateNew in interface ValueVector

allocateNewSafe

public boolean allocateNewSafe()

Allocate memory for the vector. We internally use a default value count of 4096 to allocate memory for at least these many elements in the vector. See allocateNew(int, int) for allocating memory for specific number of elements in the vector.

Specified by:

allocateNewSafe in interface ValueVector

Returns:

false if memory allocation fails, true otherwise.

allocateNew

public void allocateNew(int totalBytes,
               int valueCount)

Allocate memory for the vector to support storing at least the provided number of elements in the vector. This method must be called prior to using the ValueVector.

Specified by:

allocateNew in interface VariableWidthVector

Parameters:

totalBytes - desired total memory capacity

valueCount - the desired number of elements in the vector

Throws:

OutOfMemoryException

reAlloc

public void reAlloc()

Resize the vector to increase the capacity. The internal behavior is to double the current value capacity.

Specified by:

reAlloc in interface ValueVector

reallocDataBuffer

public void reallocDataBuffer()

Reallocate the data buffer. Data Buffer stores the actual data for VARCHAR or VARBINARY elements in the vector. The behavior is to double the size of buffer.

Throws:

OversizedAllocationException - if the desired new size is more than max allowed

OutOfMemoryException - if the internal memory allocation fails

reallocValidityAndOffsetBuffers

public void reallocValidityAndOffsetBuffers()

Reallocate the validity and offset buffers for this vector. Validity buffer is used to track the NULL or NON-NULL nature of elements in the vector and offset buffer is used to store the lengths of variable width elements in the vector. Note that data buffer for variable length vectors moves independent of the companion validity and offset buffers. This is in contrast to what we have for fixed width vectors. So even though we may have setup an initial capacity of 1024 elements in the vector, it is quite possible that we need to reAlloc() the data buffer when we are setting the 5th element in the vector simply because previous variable length elements have exhausted the buffer capacity. However, we really don't need to reAlloc() validity and offset buffers until we try to set the 1025th element This is why we do a separate check for safe methods to determine which buffer needs reallocation.

Throws:

OversizedAllocationException - if the desired new size is more than max allowed

OutOfMemoryException - if the internal memory allocation fails

getByteCapacity

public int getByteCapacity()

Get the size (number of bytes) of underlying data buffer.

Specified by:

getByteCapacity in interface VariableWidthVector

Returns:

getCurrentSizeInBytes

public int getCurrentSizeInBytes()

Specified by:

getCurrentSizeInBytes in interface VariableWidthVector

getBufferSize

public int getBufferSize()

Get the size (number of bytes) of underlying buffers used by this vector

Specified by:

getBufferSize in interface ValueVector

Returns:

size of underlying buffers.

getBufferSizeFor

public int getBufferSizeFor(int valueCount)

Get the potential buffer size for a particular number of records.

Specified by:

getBufferSizeFor in interface ValueVector

Parameters:

valueCount - desired number of elements in the vector

Returns:

estimated size of underlying buffers if the vector holds a given number of elements

getField

public Field getField()

Get information about how this field is materialized.

Specified by:

getField in interface ValueVector

Returns:

the field corresponding to this vector

getBuffers

public ArrowBuf[] getBuffers(boolean clear)

Return the underlying buffers associated with this vector. Note that this doesn't impact the reference counts for this buffer so it only should be used for in-context access. Also note that this buffer changes regularly thus external classes shouldn't hold a reference to it (unless they change it).

Specified by:

getBuffers in interface ValueVector

Parameters:

clear - Whether to clear vector before returning; the buffers will still be refcounted but the returned array will be the only reference to them

Returns:

The underlying buffers that is used by this vector instance.

getTransferPair

public TransferPair getTransferPair(String ref,
                           BufferAllocator allocator,
                           CallBack callBack)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

callBack -

Returns:

TransferPair

getTransferPair

public TransferPair getTransferPair(BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Overrides:

getTransferPair in class BaseValueVector

Parameters:

allocator - allocator for the target vector

Returns:

TransferPair

getTransferPair

public abstract TransferPair getTransferPair(String ref,
                           BufferAllocator allocator)

Construct a transfer pair of this vector and another vector of same type.

Specified by:

getTransferPair in interface ValueVector

Parameters:

ref - name of the target vector

allocator - allocator for the target vector

Returns:

TransferPair

transferTo

public void transferTo(BaseVariableWidthVector target)

Transfer this vector'data to another vector. The memory associated with this vector is transferred to the allocator of target vector for accounting and management purposes.

Parameters:

target - destination vector for transfer

splitAndTransferTo

public void splitAndTransferTo(int startIndex,
                      int length,
                      BaseVariableWidthVector target)

Slice this vector at desired index and length and transfer the corresponding data to the target vector.

Parameters:

startIndex - start position of the split in source vector.

length - length of the split.

target - destination vector

getNullCount

public int getNullCount()

Get the number of elements that are null in the vector

Specified by:

getNullCount in interface ValueVector

Returns:

the number of null elements.

isSafe

public boolean isSafe(int index)

Check if the given index is within the current value capacity of the vector

Parameters:

index - position to check

Returns:

true if index is within the current value capacity

isNull

public boolean isNull(int index)

Check if element at given index is null.

Specified by:

isNull in interface ValueVector

Parameters:

index - position of element

Returns:

true if element at given index is null

isSet

public int isSet(int index)

Same as isNull(int).

Parameters:

index - position of element

Returns:

1 if element at given index is not null, 0 otherwise

getValueCount

public int getValueCount()

Get the value count of vector. This will always be zero unless setValueCount(int) has been called prior to calling this.

Specified by:

getValueCount in interface ValueVector

Returns:

valueCount for the vector

setValueCount

public void setValueCount(int valueCount)

Sets the value count for the vector

Specified by:

setValueCount in interface ValueVector

Parameters:

valueCount - value count

fillEmpties

public void fillEmpties(int index)

Create holes in the vector upto the given index (exclusive). Holes will be created from the current last set position in the vector.

Parameters:

index - target index

setLastSet

public void setLastSet(int value)

Set the index of last non-null element in the vector. It is important to call this method with appropriate value before calling setValueCount(int).

Parameters:

value - desired index of last non-null element.

getLastSet

public int getLastSet()

Get the index of last non-null element in the vector.

Returns:

index of the last non-null element

getStartEnd

public long getStartEnd(int index)

Get the starting position (offset) in the data stream for a given element in the vector.

Parameters:

index - position of the element in the vector

Returns:

starting offset for the element

setIndexDefined

public void setIndexDefined(int index)

Mark the particular position in the vector as non-null.

Specified by:

setIndexDefined in interface VectorDefinitionSetter

Parameters:

index - position of the element.

setValueLengthSafe

public void setValueLengthSafe(int index,
                      int length)

Sets the value length for an element.

Parameters:

index - position of the element to set

length - length of the element

getValueLength

public int getValueLength(int index)

Get the variable length element at specified index as Text.

Parameters:

index - position of element to get

Returns:

greater than 0 length for non-null element, 0 otherwise

set

public void set(int index,
       byte[] value)

Set the variable length element at the specified index to the supplied byte array. This is same as using set(int, byte[], int, int) with start as 0 and length as value.length

Parameters:

index - position of the element to set

value - array of bytes to write

setSafe

public void setSafe(int index,
           byte[] value)

Same as set(int, byte[]) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.

Parameters:

index - position of the element to set

value - array of bytes to write

set

public void set(int index,
       byte[] value,
       int start,
       int length)

Set the variable length element at the specified index to the supplied byte array.

Parameters:

index - position of the element to set

value - array of bytes to write

start - start index in array of bytes

length - length of data in array of bytes

setSafe

public void setSafe(int index,
           byte[] value,
           int start,
           int length)

Same as set(int, byte[], int, int) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.

Parameters:

index - position of the element to set

value - array of bytes to write

start - start index in array of bytes

length - length of data in array of bytes

set

public void set(int index,
       ByteBuffer value,
       int start,
       int length)

Set the variable length element at the specified index to the content in supplied ByteBuffer

Parameters:

index - position of the element to set

value - ByteBuffer with data

start - start index in ByteBuffer

length - length of data in ByteBuffer

setSafe

public void setSafe(int index,
           ByteBuffer value,
           int start,
           int length)

Same as set(int, ByteBuffer, int, int) except that it handles the case where index and length of new element are beyond the existing capacity of the vector.

Parameters:

index - position of the element to set

value - ByteBuffer with data

start - start index in ByteBuffer

length - length of data in ByteBuffer

setNull

public void setNull(int index)

Set the element at the given index to null.

Parameters:

index - position of element

set

public void set(int index,
       int isSet,
       int start,
       int end,
       ArrowBuf buffer)

Store the given value at a particular position in the vector. isSet indicates whether the value is NULL or not.

Parameters:

index - position of the new value

isSet - 0 for NULL value, 1 otherwise

start - start position of data in buffer

end - end position of data in buffer

buffer - data buffer containing the variable width element to be stored in the vector

setSafe

public void setSafe(int index,
           int isSet,
           int start,
           int end,
           ArrowBuf buffer)

Same as set(int, int, int, int, ArrowBuf) except that it handles the case when index is greater than or equal to current value capacity of the vector.

Parameters:

index - position of the new value

isSet - 0 for NULL value, 1 otherwise

start - start position of data in buffer

end - end position of data in buffer

buffer - data buffer containing the variable width element to be stored in the vector

set

public void set(int index,
       int start,
       int length,
       ArrowBuf buffer)

Store the given value at a particular position in the vector. isSet indicates whether the value is NULL or not.

Parameters:

index - position of the new value

start - start position of data in buffer

length - length of data in buffer

buffer - data buffer containing the variable width element to be stored in the vector

setSafe

public void setSafe(int index,
           int start,
           int length,
           ArrowBuf buffer)

Same as set(int, int, int, int, ArrowBuf) except that it handles the case when index is greater than or equal to current value capacity of the vector.

Parameters:

index - position of the new value

start - start position of data in buffer

length - length of data in buffer

buffer - data buffer containing the variable width element to be stored in the vector

fillHoles

protected final void fillHoles(int index)

* helper methods for setters * *

setBytes

protected final void setBytes(int index,
            byte[] value,
            int start,
            int length)

getstartOffset

protected final int getstartOffset(int index)

handleSafe

protected final void handleSafe(int index,
              int dataLength)

get

public static byte[] get(ArrowBuf data,
         ArrowBuf offset,
         int index)

Method used by Json Writer to read a variable width element from the variable width vector and write to Json. This method should not be used externally.

Parameters:

data - buffer storing the variable width vector elements

offset - buffer storing the offsets of variable width vector elements

index - position of the element in the vector

Returns:

array of bytes

set

public static ArrowBuf set(ArrowBuf buffer,
           BufferAllocator allocator,
           int valueCount,
           int index,
           int value)

Method used by Json Reader to explicitly set the offsets of the variable width vector data. The method takes care of allocating the memory for offsets if the caller hasn't done so. This method should not be used externally.

Parameters:

buffer - ArrowBuf to store offsets for variable width elements

allocator - memory allocator

valueCount - number of elements

index - position of the element

value - offset of the element

Returns:

buffer holding the offsets