org.deeplearning4j.models.word2vec.iterator

Class Word2VecDataSetIterator

java.lang.Object

org.deeplearning4j.models.word2vec.iterator.Word2VecDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class Word2VecDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Iterates over a sentence with moving window to produce a data applyTransformToDestination for word windows based on a pretrained word2vec.

Author:

Adam Gibson

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels) Initializes this iterator with homogenization and adding labels and a batch size of 10
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels, int batch) Initializes this iterator with homogenization and adding labels
Word2VecDataSetIterator(Word2Vec vec, LabelAwareSentenceIterator iter, List<String> labels, int batch, boolean homogenization, boolean addLabels) Allows for customization of all of the params of the iterator

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
int	cursor()
List<String>	getLabels()
boolean	hasNext() Returns true if the iteration has more elements.
int	inputColumns()
org.nd4j.linalg.dataset.DataSet	next() Returns the next element in the iteration.
org.nd4j.linalg.dataset.DataSet	next(int num) Like the standard next method but allows a customizable number of examples returned
int	numExamples()
void	remove() Removes from the underlying collection the last element returned by this iterator (optional operation).
void	reset()
boolean	resetSupported()
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
int	totalExamples()
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels,
                               int batch,
                               boolean homogenization,
                               boolean addLabels)

Allows for customization of all of the params of the iterator

Parameters:

vec - the word2vec model to use

iter - the sentence iterator to use

labels - the possible labels

batch - the batch size

homogenization - whether to homogenize the sentences or not

addLabels - whether to add labels for windows

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels)

Initializes this iterator with homogenization and adding labels and a batch size of 10

Parameters:

vec - the vector model to use

iter - the sentence iterator to use

labels - the possible labels

Word2VecDataSetIterator

public Word2VecDataSetIterator(Word2Vec vec,
                               LabelAwareSentenceIterator iter,
                               List<String> labels,
                               int batch)

Initializes this iterator with homogenization and adding labels

Parameters:

vec - the vector model to use

iter - the sentence iterator to use

labels - the possible labels

batch - the batch size

Method Detail

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Like the standard next method but allows a customizable number of examples returned

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

Parameters:

num - the number of examples

Returns:

the next data applyTransformToDestination

totalExamples

public int totalExamples()

Specified by:

totalExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

cursor

public int cursor()

Specified by:

cursor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

numExamples

public int numExamples()

Specified by:

numExamples in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Returns true if the iteration has more elements. (In other words, returns true if next(int) would return an element rather than throwing an exception.)

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Returns:

true if the iteration has more elements

next

public org.nd4j.linalg.dataset.DataSet next()

Returns the next element in the iteration.

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Returns:

the next element in the iteration

remove

public void remove()

Removes from the underlying collection the last element returned by this iterator (optional operation). This method can be called only once per call to next(int). The behavior of an iterator is unspecified if the underlying collection is modified while the iteration is in progress in any way other than by calling this method.

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>

Throws:

UnsupportedOperationException - if the remove operation is not supported by this iterator

IllegalStateException - if the next method has not yet been called, or the remove method has already been called after the last call to the next method