org.datavec.hadoop.records.reader.mapfile

Class MapFileReader<V>

java.lang.Object

org.datavec.hadoop.records.reader.mapfile.MapFileReader<V>

All Implemented Interfaces:

Closeable, AutoCloseable

public class MapFileReader<V>
extends Object
implements Closeable

A wrapper around a Hadoop MapFile.Reader, used in MapFileRecordReader and MapFileSequenceRecordReader Note: This also handles multiple map files, such as the output from Spark, which gives a set of map files in directories like /part-r-00000, /part-r-00001

Author:

Alex Black

Constructor Summary

Constructors

Constructor and Description
MapFileReader(List<String> paths, IndexToKey indexToKey, Class<? extends org.apache.hadoop.io.Writable> recordClass)
MapFileReader(String path)
MapFileReader(String path, IndexToKey indexToKey, Class<? extends org.apache.hadoop.io.Writable> recordClass)

Method Summary

Modifier and Type	Method and Description
void	close()
V	getRecord(long index) It a single record from the map file for the given index
long	numRecords() Determine the total number of records in the map file, using the IndexToKey instance

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MapFileReader

public MapFileReader(String path)
              throws Exception

Throws:

Exception

MapFileReader

public MapFileReader(String path,
                     IndexToKey indexToKey,
                     Class<? extends org.apache.hadoop.io.Writable> recordClass)
              throws IOException

Parameters:

path - Path (directory) of the MapFile

indexToKey - Instance used to convert long indices to key values. This allows for lookup by key

recordClass - Class of the records in the MapFile

Throws:

IOException - If an error occurs during opening or initialisation

MapFileReader

public MapFileReader(List<String> paths,
                     IndexToKey indexToKey,
                     Class<? extends org.apache.hadoop.io.Writable> recordClass)
              throws IOException

Throws:

IOException

Method Detail

numRecords

public long numRecords()

Determine the total number of records in the map file, using the IndexToKey instance

Returns:

Total number of records and the map file

getRecord

public V getRecord(long index)
            throws IOException

It a single record from the map file for the given index

Parameters:

index - Index, between 0 and numRecords()-1

Returns:

Value from the MapFile

Throws:

IOException - If an error occurs during reading

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException