org.datavec.hadoop.records.writer.mapfile

Class MapFileSequenceRecordWriter

java.lang.Object

org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter<List<List<Writable>>>

org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, SequenceRecordWriter

public class MapFileSequenceRecordWriter
extends AbstractMapFileWriter<List<List<Writable>>>
implements SequenceRecordWriter

MapFileSequenceRecordWriter is used to write sequence values to a Hadoop MapFile, that can then be read by: MapFileSequenceRecordReader

Author:

Alex Black

See Also:

MapFileSequenceRecordReader

Field Summary

Fields inherited from class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

convertTextTo, counter, DEFAULT_FILENAME_PATTERN, DEFAULT_INDEX_INTERVAL, DEFAULT_MAP_FILE_SPLIT_SIZE, filenamePattern, hadoopConfiguration, indexInterval, isClosed, KEY_CLASS, MAP_FILE_INDEX_INTERVAL_KEY, mapFileSplitSize, opts, outputDir, outputFiles, writers

Fields inherited from interface org.datavec.api.records.writer.SequenceRecordWriter

APPEND

Constructor Summary

Constructors

Constructor and Description
MapFileSequenceRecordWriter(File outputDir) Constructor for all default values.
MapFileSequenceRecordWriter(File outputDir, int mapFileSplitSize) Constructor for most default values.
MapFileSequenceRecordWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo)
MapFileSequenceRecordWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo, org.apache.hadoop.conf.Configuration hadoopConfiguration)
MapFileSequenceRecordWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo, int indexInterval, org.apache.hadoop.conf.Configuration hadoopConfiguration)
MapFileSequenceRecordWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo, int indexInterval, String filenamePattern, org.apache.hadoop.conf.Configuration hadoopConfiguration)
MapFileSequenceRecordWriter(File outputDir, WritableType convertTextTo)

Method Summary

Modifier and Type	Method and Description
protected org.apache.hadoop.io.Writable	getHadoopWritable(List<List<Writable>> input)
protected Class<? extends org.apache.hadoop.io.Writable>	getValueClass()

Methods inherited from class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

close, convertTextWritables, getConf, setConf, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.records.writer.SequenceRecordWriter

close, write

Methods inherited from interface org.datavec.api.conf.Configurable

getConf, setConf

Constructor Detail

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(File outputDir)

Constructor for all default values. Single output MapFile, no text writable conversion, default index interval (1), default naming pattern.

Parameters:

outputDir - Output directory for the map file(s)

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   int mapFileSplitSize)

Constructor for most default values. Specified number of output MapFile s, no text writable conversion, default index interval (1), default naming pattern.

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize examples. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example).

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   WritableType convertTextTo)

Parameters:

outputDir - Output directory for the map file(s)

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   int mapFileSplitSize,
                                   WritableType convertTextTo)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize examples. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example).

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   int mapFileSplitSize,
                                   WritableType convertTextTo,
                                   org.apache.hadoop.conf.Configuration hadoopConfiguration)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize examples. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example).

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

hadoopConfiguration - Hadoop configuration.

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   int mapFileSplitSize,
                                   WritableType convertTextTo,
                                   int indexInterval,
                                   org.apache.hadoop.conf.Configuration hadoopConfiguration)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize examples. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example).

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most cases

hadoopConfiguration - Hadoop configuration.

MapFileSequenceRecordWriter

public MapFileSequenceRecordWriter(@NonNull
                                   File outputDir,
                                   int mapFileSplitSize,
                                   WritableType convertTextTo,
                                   int indexInterval,
                                   String filenamePattern,
                                   org.apache.hadoop.conf.Configuration hadoopConfiguration)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize examples. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example).

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most cases

filenamePattern - The naming pattern for the map files. Used with String.format(pattern, int)

hadoopConfiguration - Hadoop configuration.

Method Detail

getValueClass

protected Class<? extends org.apache.hadoop.io.Writable> getValueClass()

Specified by:

getValueClass in class AbstractMapFileWriter<List<List<Writable>>>

getHadoopWritable

protected org.apache.hadoop.io.Writable getHadoopWritable(List<List<Writable>> input)

Specified by:

getHadoopWritable in class AbstractMapFileWriter<List<List<Writable>>>