org.datavec.api.records.reader.impl.misc

Class SVMLightRecordReader

java.lang.Object

org.datavec.api.records.reader.BaseRecordReader

org.datavec.api.records.reader.impl.LineRecordReader

org.datavec.api.records.reader.impl.misc.SVMLightRecordReader

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, Configurable, RecordReader

Direct Known Subclasses:

LibSvmRecordReader

public class SVMLightRecordReader
extends LineRecordReader

Record reader for SVMLight format, which can generally be described as LABEL INDEX:VALUE INDEX:VALUE ... SVMLight format is well-suited to sparse data (e.g., bag-of-words) because it omits all features with value zero. We support an "extended" version that allows for multiple targets (or labels) separated by a comma, as follows: LABEL1,LABEL2,... INDEX:VALUE INDEX:VALUE ... This can be used to represent either multitask problems or multilabel problems with sparse binary labels (controlled via the "MULTILABEL" configuration option). Like scikit-learn, we support both zero-based and one-based indexing. Further details on the format can be found at - http://svmlight.joachims.org/ - http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multilabel.html - http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_svmlight_file.html

Author:

Adam Gibson (original), Josh Patterson, dave@skymind.io

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	ALLOWED_DELIMITERS
protected boolean	appendLabel
static String	COMMENT_CHAR
static String	FEATURE_DELIMITER
static String	LABEL_DELIMITER
static Writable	LABEL_ONE
static Writable	LABEL_ZERO
protected boolean	multilabel
static String	MULTILABEL
static String	NAME_SPACE
static String	NUM_ATTRIBUTES
static String	NUM_FEATURES
static String	NUM_LABELS
protected int	numFeatures
protected int	numLabels
static Writable	ONE
static String	PREFERRED_DELIMITER
static String	QID_PREFIX
protected Writable	recordLookahead
static Writable	ZERO
static String	ZERO_BASED_INDEXING
static String	ZERO_BASED_LABEL_INDEXING
protected boolean	zeroBasedIndexing
protected boolean	zeroBasedLabelIndexing

Fields inherited from class org.datavec.api.records.reader.impl.LineRecordReader

conf, inputSplit, lineIndex, locations, splitIndex

Fields inherited from class org.datavec.api.records.reader.BaseRecordReader

listeners

Fields inherited from interface org.datavec.api.records.reader.RecordReader

APPEND_LABEL, LABELS

Constructor Summary

Constructors

Constructor and Description
SVMLightRecordReader()

Method Summary

Modifier and Type	Method and Description
protected Writable	getNextRecord() Helper function to help detect lines that are commented out.
boolean	hasNext() Whether there are anymore records
void	initialize(Configuration conf, InputSplit split) Must be called before attempting to read records.
List<Writable>	next() Return next record as list of Writables.
Record	nextRecord() Return next Record.
protected void	onLocationOpen(URI location)
List<Writable>	record(URI uri, DataInputStream dataInputStream) Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream
void	reset() Reset record reader iterator
void	setConf(Configuration conf) Set configuration.

Methods inherited from class org.datavec.api.records.reader.impl.LineRecordReader

close, closeIfRequired, getConf, getIterator, getLabels, initialize, loadFromMetaData, loadFromMetaData

Methods inherited from class org.datavec.api.records.reader.BaseRecordReader

batchesSupported, getListeners, invokeListeners, next, setListeners, setListeners

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME_SPACE

public static final String NAME_SPACE

NUM_FEATURES

public static final String NUM_FEATURES

ZERO_BASED_INDEXING

public static final String ZERO_BASED_INDEXING

ZERO_BASED_LABEL_INDEXING

public static final String ZERO_BASED_LABEL_INDEXING

MULTILABEL

public static final String MULTILABEL

NUM_LABELS

public static final String NUM_LABELS

COMMENT_CHAR

public static final String COMMENT_CHAR

See Also:

Constant Field Values

ALLOWED_DELIMITERS

public static final String ALLOWED_DELIMITERS

See Also:

Constant Field Values

PREFERRED_DELIMITER

public static final String PREFERRED_DELIMITER

See Also:

Constant Field Values

FEATURE_DELIMITER

public static final String FEATURE_DELIMITER

See Also:

Constant Field Values

LABEL_DELIMITER

public static final String LABEL_DELIMITER

See Also:

Constant Field Values

QID_PREFIX

public static final String QID_PREFIX

See Also:

Constant Field Values

ZERO

public static final Writable ZERO

ONE

public static final Writable ONE

LABEL_ZERO

public static final Writable LABEL_ZERO

LABEL_ONE

public static final Writable LABEL_ONE

numFeatures

protected int numFeatures

zeroBasedIndexing

protected boolean zeroBasedIndexing

zeroBasedLabelIndexing

protected boolean zeroBasedLabelIndexing

appendLabel

protected boolean appendLabel

multilabel

protected boolean multilabel

numLabels

protected int numLabels

recordLookahead

protected Writable recordLookahead

NUM_ATTRIBUTES

public static final String NUM_ATTRIBUTES

Constructor Detail

SVMLightRecordReader

public SVMLightRecordReader()

Method Detail

initialize

public void initialize(Configuration conf,
                       InputSplit split)
                throws IOException,
                       InterruptedException

Must be called before attempting to read records.

Specified by:

initialize in interface RecordReader

Overrides:

initialize in class LineRecordReader

Parameters:

conf - DataVec configuration

split - FileSplit

Throws:

IOException

InterruptedException

setConf

public void setConf(Configuration conf)

Set configuration.

Specified by:

setConf in interface Configurable

Overrides:

setConf in class LineRecordReader

Parameters:

conf - DataVec configuration

Throws:

IOException

InterruptedException

getNextRecord

protected Writable getNextRecord()

Helper function to help detect lines that are commented out. May read ahead and cache a line.

Returns:

hasNext

public boolean hasNext()

Description copied from interface: RecordReader

Whether there are anymore records

Specified by:

hasNext in interface RecordReader

Overrides:

hasNext in class LineRecordReader

Returns:

next

public List<Writable> next()

Return next record as list of Writables.

Specified by:

next in interface RecordReader

Overrides:

next in class LineRecordReader

Returns:

nextRecord

public Record nextRecord()

Return next Record.

Specified by:

nextRecord in interface RecordReader

Overrides:

nextRecord in class LineRecordReader

Returns:

record

public List<Writable> record(URI uri,
                             DataInputStream dataInputStream)
                      throws IOException

Description copied from interface: RecordReader

Load the record from the given DataInputStream Unlike RecordReader.next() the internal state of the RecordReader is not modified Implementations of this method should not close the DataInputStream

Specified by:

record in interface RecordReader

Overrides:

record in class LineRecordReader

Throws:

IOException - if error occurs during reading from the input stream

reset

public void reset()

Description copied from interface: RecordReader

Reset record reader iterator

Specified by:

reset in interface RecordReader

Overrides:

reset in class LineRecordReader

onLocationOpen

protected void onLocationOpen(URI location)

Overrides:

onLocationOpen in class LineRecordReader