org.datavec.hadoop.records.writer.mapfile

Class AbstractMapFileWriter<T>

java.lang.Object

org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter<T>

Direct Known Subclasses:

MapFileRecordWriter, MapFileSequenceRecordWriter

public abstract class AbstractMapFileWriter<T>
extends Object

An abstract class For creating Hadoop map files, that underlies MapFileRecordWriter and MapFileSequenceRecordWriter.

Author:

Alex Black

Field Summary

Fields

Modifier and Type	Field and Description
protected WritableType	convertTextTo
protected AtomicLong	counter
static String	DEFAULT_FILENAME_PATTERN
static int	DEFAULT_INDEX_INTERVAL
static int	DEFAULT_MAP_FILE_SPLIT_SIZE
protected String	filenamePattern
protected org.apache.hadoop.conf.Configuration	hadoopConfiguration
protected int	indexInterval
protected AtomicBoolean	isClosed
static Class<? extends org.apache.hadoop.io.WritableComparable>	KEY_CLASS
static String	MAP_FILE_INDEX_INTERVAL_KEY Configuration key for the map file interval.
protected int	mapFileSplitSize
protected org.apache.hadoop.io.SequenceFile.Writer.Option[]	opts
protected File	outputDir
protected List<File>	outputFiles
protected List<org.apache.hadoop.io.MapFile.Writer>	writers

Constructor Summary

Constructors

Constructor and Description
AbstractMapFileWriter(File outputDir) Constructor for all default values.
AbstractMapFileWriter(File outputDir, int mapFileSplitSize) Constructor for most default values.
AbstractMapFileWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo)
AbstractMapFileWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo, int indexInterval, org.apache.hadoop.conf.Configuration hadoopConfiguration)
AbstractMapFileWriter(File outputDir, int mapFileSplitSize, WritableType convertTextTo, int indexInterval, String filenamePattern, org.apache.hadoop.conf.Configuration hadoopConfiguration)
AbstractMapFileWriter(File outputDir, WritableType convertTextTo)

Method Summary

Modifier and Type	Method and Description
void	close()
protected List<Writable>	convertTextWritables(List<Writable> record)
Configuration	getConf()
protected abstract org.apache.hadoop.io.Writable	getHadoopWritable(T input)
protected abstract Class<? extends org.apache.hadoop.io.Writable>	getValueClass()
void	setConf(Configuration conf)
void	write(T record)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_FILENAME_PATTERN

public static final String DEFAULT_FILENAME_PATTERN

See Also:

Constant Field Values

KEY_CLASS

public static final Class<? extends org.apache.hadoop.io.WritableComparable> KEY_CLASS

MAP_FILE_INDEX_INTERVAL_KEY

public static final String MAP_FILE_INDEX_INTERVAL_KEY

Configuration key for the map file interval. This is defined in MapFile.Writer.INDEX_INTERVAL but unfortunately that field is private, hence cannot be referenced here.

See Also:

Constant Field Values

DEFAULT_MAP_FILE_SPLIT_SIZE

public static final int DEFAULT_MAP_FILE_SPLIT_SIZE

See Also:

Constant Field Values

DEFAULT_INDEX_INTERVAL

public static final int DEFAULT_INDEX_INTERVAL

See Also:

Constant Field Values

outputDir

protected final File outputDir

mapFileSplitSize

protected final int mapFileSplitSize

convertTextTo

protected final WritableType convertTextTo

indexInterval

protected final int indexInterval

filenamePattern

protected final String filenamePattern

hadoopConfiguration

protected org.apache.hadoop.conf.Configuration hadoopConfiguration

counter

protected final AtomicLong counter

isClosed

protected final AtomicBoolean isClosed

outputFiles

protected List<File> outputFiles

writers

protected List<org.apache.hadoop.io.MapFile.Writer> writers

opts

protected org.apache.hadoop.io.SequenceFile.Writer.Option[] opts

Constructor Detail

AbstractMapFileWriter

public AbstractMapFileWriter(File outputDir)

Constructor for all default values. Single output MapFile, no text writable conversion, default index interval (1), default naming pattern.

Parameters:

outputDir - Output directory for the map file(s)

AbstractMapFileWriter

public AbstractMapFileWriter(@NonNull
                             File outputDir,
                             int mapFileSplitSize)

Constructor for most default values. Specified number of output MapFile s, no text writable conversion, default index interval (1), default naming pattern.

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example)

AbstractMapFileWriter

public AbstractMapFileWriter(@NonNull
                             File outputDir,
                             WritableType convertTextTo)

Parameters:

outputDir - Output directory for the map file(s)

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

AbstractMapFileWriter

public AbstractMapFileWriter(@NonNull
                             File outputDir,
                             int mapFileSplitSize,
                             WritableType convertTextTo)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example)

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

AbstractMapFileWriter

public AbstractMapFileWriter(@NonNull
                             File outputDir,
                             int mapFileSplitSize,
                             WritableType convertTextTo,
                             int indexInterval,
                             org.apache.hadoop.conf.Configuration hadoopConfiguration)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example)

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most cases

hadoopConfiguration - Hadoop configuration.

AbstractMapFileWriter

public AbstractMapFileWriter(@NonNull
                             File outputDir,
                             int mapFileSplitSize,
                             WritableType convertTextTo,
                             int indexInterval,
                             String filenamePattern,
                             org.apache.hadoop.conf.Configuration hadoopConfiguration)

Parameters:

outputDir - Output directory for the map file(s)

mapFileSplitSize - Split size for the map file: if 0, use a single map file for all output. If > 0, multiple map files will be used: each will contain a maximum of mapFileSplitSize. This can be used to avoid having a single multi gigabyte map file, which may be undesirable in some cases (transfer across the network, for example)

convertTextTo - If null: Make no changes to Text writable objects. If non-null, Text writable instances will be converted to this type. This is useful, when would rather store numerical values even if the original record reader produces strings/text.

indexInterval - Index interval for the Map file. Defaults to 1, which is suitable for most cases

filenamePattern - The naming pattern for the map files. Used with String.format(pattern, int)

hadoopConfiguration - Hadoop configuration.

Method Detail

getValueClass

protected abstract Class<? extends org.apache.hadoop.io.Writable> getValueClass()

setConf

public void setConf(Configuration conf)

getConf

public Configuration getConf()

getHadoopWritable

protected abstract org.apache.hadoop.io.Writable getHadoopWritable(T input)

convertTextWritables

protected List<Writable> convertTextWritables(List<Writable> record)

write

public void write(T record)
           throws IOException

Throws:

IOException

close

public void close()