org.apache.arrow.vector.ipc.message

Class MessageSerializer

java.lang.Object

org.apache.arrow.vector.ipc.message.MessageSerializer

public class MessageSerializer
extends Object

Utility class for serializing Messages. Messages are all serialized a similar way. 1. 4 byte little endian message header prefix 2. FB serialized Message: This includes it the body length, which is the serialized body and the type of the message. 3. Serialized message. For schema messages, the serialization is simply the FB serialized Schema. For RecordBatch messages the serialization is: 1. 4 byte little endian batch metadata header 2. FB serialized RowBatch 3. Padding to align to 8 byte boundary. 4. serialized RowBatch buffers.

Constructor Summary

Constructors

Constructor and Description
MessageSerializer()

Method Summary

Methods

Modifier and Type	Method and Description
static int	bytesToInt(byte[] bytes)
static ArrowDictionaryBatch	deserializeDictionaryBatch(MessageReader reader, Message message, BufferAllocator alloc) Deserializes a DictionaryBatch.
static ArrowDictionaryBatch	deserializeDictionaryBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) Deserializes a DictionaryBatch knowing the size of the entire message up front.
static ArrowMessage	deserializeMessageBatch(MessageReader reader, BufferAllocator alloc) Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.
static ArrowMessage	deserializeMessageBatch(ReadChannel in, BufferAllocator alloc) Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.
static ArrowRecordBatch	deserializeRecordBatch(MessageReader reader, Message message, BufferAllocator alloc) Deserializes a RecordBatch.
static ArrowRecordBatch	deserializeRecordBatch(ReadChannel in, ArrowBlock block, BufferAllocator alloc) Deserializes a RecordBatch knowing the size of the entire message up front.
static ArrowRecordBatch	deserializeRecordBatch(RecordBatch recordBatchFB, ArrowBuf body) Deserializes a record batch given the Flatbuffer metadata and in-memory body.
static Schema	deserializeSchema(MessageReader reader) Deserializes a schema object.
static Schema	deserializeSchema(ReadChannel in) Deserializes a schema object.
static ArrowBlock	serialize(WriteChannel out, ArrowDictionaryBatch batch) Serializes a dictionary ArrowRecordBatch.
static ArrowBlock	serialize(WriteChannel out, ArrowRecordBatch batch) Serializes an ArrowRecordBatch.
static long	serialize(WriteChannel out, Schema schema) Serialize a schema object.
static ByteBuffer	serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder, byte headerType, int headerOffset, int bodyLength) Serializes a message header.
static long	writeBatchBuffers(WriteChannel out, ArrowRecordBatch batch)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MessageSerializer

public MessageSerializer()

Method Detail

bytesToInt

public static int bytesToInt(byte[] bytes)

serialize

public static long serialize(WriteChannel out,
             Schema schema)
                      throws IOException

Serialize a schema object.

Parameters:

out - where to write the schema

schema - the object to serialize to out

Returns:

the resulting size of the serialized schema

Throws:

IOException - if something went wrong

deserializeSchema

public static Schema deserializeSchema(MessageReader reader)
                                throws IOException

Deserializes a schema object. Format is from serialize().

Parameters:

reader - the reader interface to deserialize from

Returns:

the deserialized object

Throws:

IOException - if something went wrong

deserializeSchema

public static Schema deserializeSchema(ReadChannel in)
                                throws IOException

Deserializes a schema object. Format is from serialize().

Parameters:

in - the channel to deserialize from

Returns:

the deserialized object

Throws:

IOException - if something went wrong

serialize

public static ArrowBlock serialize(WriteChannel out,
                   ArrowRecordBatch batch)
                            throws IOException

Serializes an ArrowRecordBatch. Returns the offset and length of the written batch.

Parameters:

out - where to write the batch

batch - the object to serialize to out

Returns:

the serialized block metadata

Throws:

IOException - if something went wrong

writeBatchBuffers

public static long writeBatchBuffers(WriteChannel out,
                     ArrowRecordBatch batch)
                              throws IOException

Throws:

IOException

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(MessageReader reader,
                                      Message message,
                                      BufferAllocator alloc)
                                               throws IOException

Deserializes a RecordBatch.

Parameters:

reader - the reader interface to deserialize from

message - the object to derialize to

alloc - to allocate buffers

Returns:

the deserialized object

Throws:

IOException - if something went wrong

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(ReadChannel in,
                                      ArrowBlock block,
                                      BufferAllocator alloc)
                                               throws IOException

Deserializes a RecordBatch knowing the size of the entire message up front. This minimizes the number of reads to the underlying stream.

Parameters:

in - the channel to deserialize from

block - the object to derialize to

alloc - to allocate buffers

Returns:

the deserialized object

Throws:

IOException - if something went wrong

deserializeRecordBatch

public static ArrowRecordBatch deserializeRecordBatch(RecordBatch recordBatchFB,
                                      ArrowBuf body)
                                               throws IOException

Deserializes a record batch given the Flatbuffer metadata and in-memory body.

Parameters:

recordBatchFB - Deserialized FlatBuffer record batch

body - Read body of the record batch

Returns:

ArrowRecordBatch from metadata and in-memory body

Throws:

IOException

serialize

public static ArrowBlock serialize(WriteChannel out,
                   ArrowDictionaryBatch batch)
                            throws IOException

Serializes a dictionary ArrowRecordBatch. Returns the offset and length of the written batch.

Parameters:

out - where to serialize

batch - the batch to serialize

Returns:

the metadata of the serialized block

Throws:

IOException - if something went wrong

deserializeDictionaryBatch

public static ArrowDictionaryBatch deserializeDictionaryBatch(MessageReader reader,
                                              Message message,
                                              BufferAllocator alloc)
                                                       throws IOException

Deserializes a DictionaryBatch.

Parameters:

reader - where to read from

message - the message message metadata to deserialize

alloc - the allocator for new buffers

Returns:

the corresponding dictionary batch

Throws:

IOException - if something went wrong

deserializeDictionaryBatch

public static ArrowDictionaryBatch deserializeDictionaryBatch(ReadChannel in,
                                              ArrowBlock block,
                                              BufferAllocator alloc)
                                                       throws IOException

Deserializes a DictionaryBatch knowing the size of the entire message up front. This minimizes the number of reads to the underlying stream.

Parameters:

in - where to read from

block - block metadata for deserializing

alloc - to allocate new buffers

Returns:

the corresponding dictionary

Throws:

IOException - if something went wrong

deserializeMessageBatch

public static ArrowMessage deserializeMessageBatch(MessageReader reader,
                                   BufferAllocator alloc)
                                            throws IOException

Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.

Parameters:

reader - Interface to read messages from

alloc - Allocator for message data

Returns:

The deserialized record batch

Throws:

IOException - if the message is not an ArrowDictionaryBatch or ArrowRecordBatch

deserializeMessageBatch

public static ArrowMessage deserializeMessageBatch(ReadChannel in,
                                   BufferAllocator alloc)
                                            throws IOException

Deserialize a message that is either an ArrowDictionaryBatch or ArrowRecordBatch.

Parameters:

in - ReadChannel to read messages from

alloc - Allocator for message data

Returns:

The deserialized record batch

Throws:

IOException - if the message is not an ArrowDictionaryBatch or ArrowRecordBatch

serializeMessage

public static ByteBuffer serializeMessage(com.google.flatbuffers.FlatBufferBuilder builder,
                          byte headerType,
                          int headerOffset,
                          int bodyLength)

Serializes a message header.

Parameters:

builder - to write the flatbuf to

headerType - headerType field

headerOffset - header offset field

bodyLength - body length field

Returns:

the corresponding ByteBuffer