org.datavec.hadoop.records.reader.mapfile

Class MapFileSequenceRecordReader

java.lang.Object

org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader, SequenceRecordReader

public class MapFileSequenceRecordReader
extends Object
implements SequenceRecordReader

A SequenceRecordReader implementation for reading from a Hadoop MapFile

A typical use case is with TransformProcess executed on Spark (perhaps Spark local), followed by non-distributed training on a single machine. For example:

  
  JavaRDD<List<List<Writable>>> myRDD = ...;
  String mapFilePath = ...;
  SparkStorageUtils.saveMapFileSequences( mapFilePath, myRDD );

  SequenceRecordReader rr = new MapFileSequenceRecordReader();
  rr.initialize( new FileSplit( new File( mapFilePath ) ) );
  //Pass to DataSetIterator or similar
  
 

Alternatively, use MapFileSequenceRecordWriter.
Note that this sequence record reader supports optional randomisation of order.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS, NAME_SPACE

Constructor Summary

Constructors

Constructor and Description
MapFileSequenceRecordReader() Create a MapFileSequenceRecordReader with no randomisation, and assuming MapFile keys are LongWritable values
MapFileSequenceRecordReader(IndexToKey indexToKey, Random rng) Create a MapFileSequenceRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys
MapFileSequenceRecordReader(Random rng) Create a MapFileSequenceRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

Method Summary

Modifier and Type	Method and Description
boolean	batchesSupported()
void	close()
Configuration	getConf()
List<String>	getLabels()
List<RecordListener>	getListeners()
boolean	hasNext()
void	initialize(Configuration conf, InputSplit split)
void	initialize(InputSplit split)
List<Record>	loadFromMetaData(List<RecordMetaData> recordMetaDatas)
Record	loadFromMetaData(RecordMetaData recordMetaData)
List<SequenceRecord>	loadSequenceFromMetaData(List<RecordMetaData> recordMetaDatas)
SequenceRecord	loadSequenceFromMetaData(RecordMetaData recordMetaData)
List<Writable>	next()
List<Writable>	next(int num)
Record	nextRecord()
SequenceRecord	nextSequence()
List<Writable>	record(URI uri, DataInputStream dataInputStream)
void	reset()
List<List<Writable>>	sequenceRecord()
List<List<Writable>>	sequenceRecord(URI uri, DataInputStream dataInputStream)
void	setConf(Configuration conf)
void	setListeners(Collection<RecordListener> listeners)
void	setListeners(RecordListener... listeners)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MapFileSequenceRecordReader

public MapFileSequenceRecordReader()

Create a MapFileSequenceRecordReader with no randomisation, and assuming MapFile keys are LongWritable values

MapFileSequenceRecordReader

public MapFileSequenceRecordReader(Random rng)

Create a MapFileSequenceRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

Parameters:

rng - If non-null, will be used to randomize the order of examples

MapFileSequenceRecordReader

public MapFileSequenceRecordReader(IndexToKey indexToKey,
                                   Random rng)

Create a MapFileSequenceRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys

Parameters:

indexToKey - Handles conversion between long indices and key values (see for example LongIndexToKey

rng - If non-null, will be used to randomize the order of examples

Method Detail

initialize

public void initialize(InputSplit split)
                throws IOException,
                       InterruptedException

Specified by:

initialize in interface RecordReader

Throws:

IOException

InterruptedException

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Specified by:

initialize in interface RecordReader

Throws:

IOException

InterruptedException

setConf

public void setConf(Configuration conf)

Specified by:

setConf in interface Configurable

getConf

public Configuration getConf()

Specified by:

getConf in interface Configurable

sequenceRecord

public List<List<Writable>> sequenceRecord()

Specified by:

sequenceRecord in interface SequenceRecordReader

sequenceRecord

public List<List<Writable>> sequenceRecord(URI uri,
                                           DataInputStream dataInputStream)
                                    throws IOException

Specified by:

sequenceRecord in interface SequenceRecordReader

Throws:

IOException

nextSequence

public SequenceRecord nextSequence()

Specified by:

nextSequence in interface SequenceRecordReader

loadSequenceFromMetaData

public SequenceRecord loadSequenceFromMetaData(@NonNull
                                               RecordMetaData recordMetaData)
                                        throws IOException

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Throws:

IOException

loadSequenceFromMetaData

public List<SequenceRecord> loadSequenceFromMetaData(@NonNull
                                                     List<RecordMetaData> recordMetaDatas)
                                              throws IOException

Specified by:

loadSequenceFromMetaData in interface SequenceRecordReader

Throws:

IOException

batchesSupported

public boolean batchesSupported()

Specified by:

batchesSupported in interface RecordReader

next

public List<Writable> next(int num)

Specified by:

next in interface RecordReader

next

public List<Writable> next()

Specified by:

next in interface RecordReader

hasNext

public boolean hasNext()

Specified by:

hasNext in interface RecordReader

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface RecordReader

reset

public void reset()

Specified by:

reset in interface RecordReader

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Specified by:

record in interface RecordReader

Throws:

IOException

nextRecord

public Record nextRecord()

Specified by:

nextRecord in interface RecordReader

loadFromMetaData

public Record loadFromMetaData(RecordMetaData recordMetaData)
                        throws IOException

Specified by:

loadFromMetaData in interface RecordReader

Throws:

IOException

loadFromMetaData

public List<Record> loadFromMetaData(List<RecordMetaData> recordMetaDatas)
                              throws IOException

Specified by:

loadFromMetaData in interface RecordReader

Throws:

IOException

getListeners

public List<RecordListener> getListeners()

Specified by:

getListeners in interface RecordReader

setListeners

public void setListeners(RecordListener... listeners)

Specified by:

setListeners in interface RecordReader

setListeners

public void setListeners(Collection<RecordListener> listeners)

Specified by:

setListeners in interface RecordReader

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException