A B C D F G H I K L M N O R S W

A

AbstractMapFileWriter<T> - Class in org.datavec.hadoop.records.writer.mapfile

An abstract class For creating Hadoop map files, that underlies MapFileRecordWriter and MapFileSequenceRecordWriter.

AbstractMapFileWriter(File) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

Constructor for all default values.

AbstractMapFileWriter(File, int) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

Constructor for most default values.

AbstractMapFileWriter(File, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

AbstractMapFileWriter(File, int, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

AbstractMapFileWriter(File, int, WritableType, int, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

AbstractMapFileWriter(File, int, WritableType, int, String, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

B

batchesSupported() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

batchesSupported() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

C

close() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileReader

close() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

close() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

close() - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

ConfigurationUtil - Class in org.datavec.hadoop.conf

Notes https://linuxjunkies.wordpress.com/2011/11/21/a-hdfsclient-for-hadoop-using-the-native-java-api-a-tutorial/ Design Ideas - Need a DataVec Conf entry: - hadoop.configuration.path - example: hadoop.configuration.path=/home/hadoop/hadoop/conf/

ConfigurationUtil() - Constructor for class org.datavec.hadoop.conf.ConfigurationUtil

convertTextTo - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

convertTextWritables(List<Writable>) - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

counter - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

createReader(InputSplit, Configuration) - Method in class org.datavec.hadoop.formats.input.SVMLightInputFormat

createReader(InputSplit) - Method in class org.datavec.hadoop.formats.input.SVMLightInputFormat

D

DEFAULT_FILENAME_PATTERN - Static variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

DEFAULT_INDEX_INTERVAL - Static variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

DEFAULT_MAP_FILE_SPLIT_SIZE - Static variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

F

filenamePattern - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

G

generateConfig(String) - Static method in class org.datavec.hadoop.conf.ConfigurationUtil

getConf() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

getConf() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

getConf() - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

getHadoopWritable(T) - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

getHadoopWritable(List<Writable>) - Method in class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

getHadoopWritable(List<List<Writable>>) - Method in class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

getKeyForIndex(long) - Method in class org.datavec.hadoop.records.reader.mapfile.index.LongIndexToKey

getKeyForIndex(long) - Method in interface org.datavec.hadoop.records.reader.mapfile.IndexToKey

Get the key for the given index

getLabels() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

getLabels() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

getListeners() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

getListeners() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

getNumRecords() - Method in class org.datavec.hadoop.records.reader.mapfile.index.LongIndexToKey

getNumRecords() - Method in interface org.datavec.hadoop.records.reader.mapfile.IndexToKey

Getter infer the number of records in the given map file(s)

getRecord(long) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileReader

It a single record from the map file for the given index

getValueClass() - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

getValueClass() - Method in class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

getValueClass() - Method in class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

H

hadoopConfiguration - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

hasNext() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

hasNext() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

hasNext() - Method in class org.datavec.hadoop.records.reader.SVMLightRecordReader

I

indexInterval - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

IndexToKey - Interface in org.datavec.hadoop.records.reader.mapfile

An interface to handle Index to key conversion, for use in MapFileReader

initialize(MapFile.Reader[], Class<? extends Writable>) - Method in class org.datavec.hadoop.records.reader.mapfile.index.LongIndexToKey

initialize(MapFile.Reader[], Class<? extends Writable>) - Method in interface org.datavec.hadoop.records.reader.mapfile.IndexToKey

Initialise the instance, and return the first and last record indexes (inclusive) for each reader

initialize(InputSplit) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

initialize(Configuration, InputSplit) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

initialize(InputSplit) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

initialize(Configuration, InputSplit) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

initialize(Configuration, InputSplit) - Method in class org.datavec.hadoop.records.reader.SVMLightRecordReader

isClosed - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

K

KEY_CLASS - Static variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

L

loadFromMetaData(RecordMetaData) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

loadFromMetaData(List<RecordMetaData>) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

loadFromMetaData(RecordMetaData) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

loadFromMetaData(List<RecordMetaData>) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

loadSequenceFromMetaData(RecordMetaData) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

loadSequenceFromMetaData(List<RecordMetaData>) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

LongIndexToKey - Class in org.datavec.hadoop.records.reader.mapfile.index

A default implementation of IndexToKey that assumes (strictly requires) keys that are LongWritable values, where all values are both unique and contiguous (0 to numRecords()-1)
This allows for easy inference of the number of records, and identify mapping between indexes and keys.

LongIndexToKey() - Constructor for class org.datavec.hadoop.records.reader.mapfile.index.LongIndexToKey

M

MAP_FILE_INDEX_INTERVAL_KEY - Static variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

Configuration key for the map file interval.

MapFileReader<V> - Class in org.datavec.hadoop.records.reader.mapfile

A wrapper around a Hadoop MapFile.Reader, used in MapFileRecordReader and MapFileSequenceRecordReader Note: This also handles multiple map files, such as the output from Spark, which gives a set of map files in directories like /part-r-00000, /part-r-00001

MapFileReader(String) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileReader

MapFileReader(String, IndexToKey, Class<? extends Writable>) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileReader

MapFileReader(List<String>, IndexToKey, Class<? extends Writable>) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileReader

MapFileRecordReader - Class in org.datavec.hadoop.records.reader.mapfile

A RecordReader implementation for reading from a Hadoop MapFile

MapFileRecordReader() - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

Create a MapFileRecordReader with no randomisation, and assuming MapFile keys are LongWritable values

MapFileRecordReader(Random) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

Create a MapFileRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

MapFileRecordReader(IndexToKey, Random) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

Create a MapFileRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys

MapFileRecordWriter - Class in org.datavec.hadoop.records.writer.mapfile

MapFileRecordWriter is used to write values to a Hadoop MapFile, that can then be read by: MapFileRecordReader

MapFileRecordWriter(File) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

Constructor for all default values.

MapFileRecordWriter(File, int) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

Constructor for most default values.

MapFileRecordWriter(File, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

MapFileRecordWriter(File, int, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

MapFileRecordWriter(File, int, WritableType, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

MapFileRecordWriter(File, int, WritableType, int, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

MapFileRecordWriter(File, int, WritableType, int, String, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileRecordWriter

MapFileSequenceRecordReader - Class in org.datavec.hadoop.records.reader.mapfile

A SequenceRecordReader implementation for reading from a Hadoop MapFile

MapFileSequenceRecordReader() - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

Create a MapFileSequenceRecordReader with no randomisation, and assuming MapFile keys are LongWritable values

MapFileSequenceRecordReader(Random) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

Create a MapFileSequenceRecordReader with optional randomisation, and assuming MapFile keys are LongWritable values

MapFileSequenceRecordReader(IndexToKey, Random) - Constructor for class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

Create a MapFileSequenceRecordReader with optional randomisation, with a custom IndexToKey instance to handle MapFile keys

MapFileSequenceRecordWriter - Class in org.datavec.hadoop.records.writer.mapfile

MapFileSequenceRecordWriter is used to write sequence values to a Hadoop MapFile, that can then be read by: MapFileSequenceRecordReader

MapFileSequenceRecordWriter(File) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

Constructor for all default values.

MapFileSequenceRecordWriter(File, int) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

Constructor for most default values.

MapFileSequenceRecordWriter(File, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

MapFileSequenceRecordWriter(File, int, WritableType) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

MapFileSequenceRecordWriter(File, int, WritableType, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

MapFileSequenceRecordWriter(File, int, WritableType, int, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

MapFileSequenceRecordWriter(File, int, WritableType, int, String, Configuration) - Constructor for class org.datavec.hadoop.records.writer.mapfile.MapFileSequenceRecordWriter

mapFileSplitSize - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

N

next(int) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

next() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

next(int) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

next() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

next() - Method in class org.datavec.hadoop.records.reader.SVMLightRecordReader

next() method for getting another K/V pair off disk from the SVMLight text file

nextRecord() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

nextRecord() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

nextSequence() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

numRecords() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileReader

Determine the total number of records in the map file, using the IndexToKey instance

O

opts - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

org.datavec.hadoop.conf - package org.datavec.hadoop.conf

org.datavec.hadoop.formats.input - package org.datavec.hadoop.formats.input

org.datavec.hadoop.records.reader - package org.datavec.hadoop.records.reader

org.datavec.hadoop.records.reader.mapfile - package org.datavec.hadoop.records.reader.mapfile

org.datavec.hadoop.records.reader.mapfile.index - package org.datavec.hadoop.records.reader.mapfile.index

org.datavec.hadoop.records.reader.mapfile.record - package org.datavec.hadoop.records.reader.mapfile.record

org.datavec.hadoop.records.writer.mapfile - package org.datavec.hadoop.records.writer.mapfile

outputDir - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

outputFiles - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

R

readFields(DataInput) - Method in class org.datavec.hadoop.records.reader.mapfile.record.RecordWritable

readFields(DataInput) - Method in class org.datavec.hadoop.records.reader.mapfile.record.SequenceRecordWritable

record(URI, DataInputStream) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

record(URI, DataInputStream) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

RecordWritable - Class in org.datavec.hadoop.records.reader.mapfile.record

Created by Alex on 29/05/2017.

RecordWritable() - Constructor for class org.datavec.hadoop.records.reader.mapfile.record.RecordWritable

reset() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

reset() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

S

sequenceRecord() - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

sequenceRecord(URI, DataInputStream) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

SequenceRecordWritable - Class in org.datavec.hadoop.records.reader.mapfile.record

Created by Alex on 29/05/2017.

SequenceRecordWritable() - Constructor for class org.datavec.hadoop.records.reader.mapfile.record.SequenceRecordWritable

setConf(Configuration) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

setConf(Configuration) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

setConf(Configuration) - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

setListeners(RecordListener...) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

setListeners(Collection<RecordListener>) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileRecordReader

setListeners(RecordListener...) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

setListeners(Collection<RecordListener>) - Method in class org.datavec.hadoop.records.reader.mapfile.MapFileSequenceRecordReader

SVMLightInputFormat - Class in org.datavec.hadoop.formats.input

SVMLight input format

SVMLightInputFormat() - Constructor for class org.datavec.hadoop.formats.input.SVMLightInputFormat

SVMLightRecordReader - Class in org.datavec.hadoop.records.reader

SVMLightRecordReader() - Constructor for class org.datavec.hadoop.records.reader.SVMLightRecordReader

W

write(DataOutput) - Method in class org.datavec.hadoop.records.reader.mapfile.record.RecordWritable

write(DataOutput) - Method in class org.datavec.hadoop.records.reader.mapfile.record.SequenceRecordWritable

write(T) - Method in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

writers - Variable in class org.datavec.hadoop.records.writer.mapfile.AbstractMapFileWriter

A B C D F G H I K L M N O R S W