org.datavec.hadoop.records.reader.mapfile

Class MapFileRecordReader

java.lang.Object

org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

public class MapFileRecordReader
extends Object
implements RecordReader

A RecordReader implementation for reading from a Hadoop MapFile

A typical use case is with TransformProcess executed on Spark (perhaps Spark local), followed by non-distributed training on a single machine. For example:

  
  JavaRDD<List<Writable>> myRDD = ...;
  String mapFilePath = ...;
  SparkStorageUtils.saveMapFile( mapFilePath, myRDD );

  RecordReader rr = new MapFileRecordReader();
  rr.initialize( new FileSplit( new File( mapFilePath ) ) );
  //Pass to DataSetIterator or similar
  
 

Alternatively, use MapFileRecordWriter.
Note that this record reader supports optional randomisation of order.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
MapFileRecordReader() Create a MapFileRecordReader with no randomisation, and assuming MapFile keys are LongWritable values
MapFileRecordReader(IndexToKey indexToKey, Random rng) Create a MapFileRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys
MapFileRecordReader(Random rng) Create a MapFileRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

Method Summary

Modifier and Type	Method and Description
boolean	batchesSupported()
void	close()
Configuration	getConf()
List<String>	getLabels()
List<RecordListener>	getListeners()
boolean	hasNext()
void	initialize(Configuration conf, InputSplit split)
void	initialize(InputSplit split)
List<Record>	loadFromMetaData(List<RecordMetaData> recordMetaDatas)
Record	loadFromMetaData(RecordMetaData recordMetaData)
List<Writable>	next()
List<Writable>	next(int num)
Record	nextRecord()
List<Writable>	record(URI uri, DataInputStream dataInputStream)
void	reset()
void	setConf(Configuration conf)
void	setListeners(Collection<RecordListener> listeners)
void	setListeners(RecordListener... listeners)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MapFileRecordReader

public MapFileRecordReader()
                    throws Exception

Create a MapFileRecordReader with no randomisation, and assuming MapFile keys are LongWritable values

Throws:

Exception

MapFileRecordReader

public MapFileRecordReader(Random rng)

Create a MapFileRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

Parameters:

rng - If non-null, will be used to randomize the order of examples

MapFileRecordReader

public MapFileRecordReader(IndexToKey indexToKey,
                           Random rng)

Create a MapFileRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys

Parameters:

indexToKey - Handles conversion between long indices and key values (see for example LongIndexToKey

rng - If non-null, will be used to randomize the order of examples

Method Detail

initialize

public void initialize(InputSplit split)
                throws IOException,
                       InterruptedException

Specified by:

initialize in interface RecordReader

Throws:

IOException

InterruptedException

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Specified by:

initialize in interface RecordReader

Throws:

IOException

InterruptedException

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

batchesSupported

public boolean batchesSupported()

Specified by:

batchesSupported in interface RecordReader

next

public List<Writable> next(int num)

Specified by:

next in interface RecordReader

next

public List<Writable> next()

Specified by:

next in interface RecordReader

hasNext

public boolean hasNext()

Specified by:

hasNext in interface RecordReader

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface RecordReader

reset

public void reset()

Specified by:

reset in interface RecordReader

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Specified by:

record in interface RecordReader

Throws:

IOException

nextRecord

public Record nextRecord()

Specified by:

nextRecord in interface RecordReader

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws IOException

Specified by:

loadFromMetaData in interface RecordReader

Throws:

IOException

loadFromMetaData

public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas)
                              throws IOException

Specified by:

loadFromMetaData in interface RecordReader

Throws:

IOException

getListeners

public List<RecordListener> getListeners()

Specified by:

getListeners in interface RecordReader

setListeners

public void setListeners(RecordListener... listeners)

Specified by:

setListeners in interface RecordReader

setListeners

public void setListeners(Collection<RecordListener> listeners)

Specified by:

setListeners in interface RecordReader

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException