org.apache.arrow.vector.stream

Class MessageSerializer

java.lang.Object

org.apache.arrow.vector.stream.MessageSerializer

public class MessageSerializer
extends Object

Utility class for serializing Messages. Messages are all serialized a similar way. 1. 4 byte little endian message header prefix 2. FB serialized Message: This includes it the body length, which is the serialized body and the type of the message. 3. Serialized message. For schema messages, the serialization is simply the FB serialized Schema. For RecordBatch messages the serialization is: 1. 4 byte little endian batch metadata header 2. FB serialized RowBatch 3. Padding to align to 8 byte boundary. 4. serialized RowBatch buffers.

Constructor Summary

Constructors

Constructor and Description
MessageSerializer()

Method Summary

Modifier and Type	Method and Description
static int	bytesToInt(byte[] bytes)
static ArrowDictionaryBatch	deserializeDictionaryBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) Deserializes a DictionaryBatch knowing the size of the entire message up front.
static ArrowDictionaryBatch	deserializeDictionaryBatch(ReadChannel in, Message message, BufferAllocator alloc) Deserializes a DictionaryBatch
static ArrowMessage	deserializeMessageBatch(ReadChannel in, BufferAllocator alloc)
static ArrowRecordBatch	deserializeRecordBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) Deserializes a RecordBatch knowing the size of the entire message up front.
static ArrowRecordBatch	deserializeRecordBatch(ReadChannel in, Message message, BufferAllocator alloc) Deserializes a RecordBatch
static ArrowRecordBatch	deserializeRecordBatch(RecordBatch recordBatchFB, ArrowBuf body)
static Schema	deserializeSchema(ReadChannel in) Deserializes a schema object.
static ArrowBlock	serialize(WriteChannel out, ArrowDictionaryBatch batch) Serializes a dictionary ArrowRecordBatch.
static ArrowBlock	serialize(WriteChannel out, ArrowRecordBatch batch) Serializes an ArrowRecordBatch.
static long	serialize(WriteChannel out, Schema schema) Serialize a schema object.
static ByteBuffer	serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder, byte headerType, int headerOffset, int bodyLength) Serializes a message header.
static long	writeBatchBuffers(WriteChannel out, ArrowRecordBatch batch)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MessageSerializer

public MessageSerializer()

Method Detail

bytesToInt

public static int bytesToInt(byte[] bytes)

serialize

public static long serialize(WriteChannel out,
                             Schema schema)
                      throws IOException

Serialize a schema object.

Parameters:

out - where to write the schema

schema - the object to serialize to out

Returns:

the resulting size of the serialized schema

Throws:

IOException - if something went wrong

deserializeSchema

public static Schema deserializeSchema(ReadChannel in)
                                throws IOException

Deserializes a schema object. Format is from serialize().

Parameters:

in - the channel to deserialize from

Returns:

the deserialized object

Throws:

IOException - if something went wrong

serialize

public static ArrowBlock serialize(WriteChannel out,
                                   ArrowRecordBatch batch)
                            throws IOException

Serializes an ArrowRecordBatch. Returns the offset and length of the written batch.

Parameters:

out - where to write the batch

batch - the object to serialize to out

Returns:

the serialized block metadata

Throws:

IOException - if something went wrong

writeBatchBuffers

public static long writeBatchBuffers(WriteChannel out,
                                     ArrowRecordBatch batch)
                              throws IOException

Throws:

IOException

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(ReadChannel in,
                                                      Message message,
                                                      BufferAllocator alloc)
                                               throws IOException

Deserializes a RecordBatch

Parameters:

in - the channel to deserialize from

message - the object to derialize to

alloc - to allocate buffers

Returns:

the deserialized object

Throws:

IOException - if something went wrong

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(ReadChannel in,
                                                      ArrowBlock block,
                                                      BufferAllocator alloc)
                                               throws IOException

Deserializes a RecordBatch knowing the size of the entire message up front. This minimizes the number of reads to the underlying stream.

Parameters:

in - the channel to deserialize from

block - the object to derialize to

alloc - to allocate buffers

Returns:

the deserialized object

Throws:

IOException - if something went wrong

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(RecordBatch recordBatchFB,
                                                      ArrowBuf body)
                                               throws IOException

Throws:

IOException

serialize

public static ArrowBlock serialize(WriteChannel out,
                                   ArrowDictionaryBatch batch)
                            throws IOException

Serializes a dictionary ArrowRecordBatch. Returns the offset and length of the written batch.

Parameters:

out - where to serialize

batch - the batch to serialize

Returns:

the metadata of the serialized block

Throws:

IOException - if something went wrong

deserializeDictionaryBatch

public static ArrowDictionaryBatch deserializeDictionaryBatch(ReadChannel in,
                                                              Message message,
                                                              BufferAllocator alloc)
                                                       throws IOException

Deserializes a DictionaryBatch

Parameters:

in - where to read from

message - the message message metadata to deserialize

alloc - the allocator for new buffers

Returns:

the corresponding dictionary batch

Throws:

IOException - if something went wrong

deserializeDictionaryBatch

public static ArrowDictionaryBatch deserializeDictionaryBatch(ReadChannel in,
                                                              ArrowBlock block,
                                                              BufferAllocator alloc)
                                                       throws IOException

Deserializes a DictionaryBatch knowing the size of the entire message up front. This minimizes the number of reads to the underlying stream.

Parameters:

in - where to read from

block - block metadata for deserializing

alloc - to allocate new buffers

Returns:

the corresponding dictionary

Throws:

IOException - if something went wrong

deserializeMessageBatch

public static ArrowMessage deserializeMessageBatch(ReadChannel in,
                                                   BufferAllocator alloc)
                                            throws IOException

Throws:

IOException

serializeMessage

public static ByteBuffer serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder,
                                          byte headerType,
                                          int headerOffset,
                                          int bodyLength)

Serializes a message header.

Parameters:

builder - to write the flatbuf to

headerType - headerType field

headerOffset - header offset field

bodyLength - body length field

Returns:

the corresponding ByteBuffer