org.deeplearning4j.models.sequencevectors.transformers.impl.iterables

Class ParallelTransformerIterator

java.lang.Object

org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.BasicTransformerIterator

org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.ParallelTransformerIterator

All Implemented Interfaces:

Iterator<Sequence<VocabWord>>

public class ParallelTransformerIterator
extends BasicTransformerIterator

TransformerIterator implementation that's does transformation/tokenization/normalization/whatever in parallel threads. Suitable for cases when tokenization takes too much time for single thread. TL/DR: we read data from sentence iterator, and apply tokenization in parallel threads.

Author:

raver119@gmail.com

Field Summary

Fields

Modifier and Type	Field and Description
protected BlockingQueue<Sequence<VocabWord>>	buffer
protected static AtomicInteger	count
protected AtomicInteger	processing
protected BlockingQueue<LabelledDocument>	stringBuffer
protected org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.ParallelTransformerIterator.TokenizerThread[]	threads
protected boolean	underlyingHas

Fields inherited from class org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.BasicTransformerIterator

allowMultithreading, iterator, sentenceTransformer

Constructor Summary

Constructors

Constructor and Description
ParallelTransformerIterator(LabelAwareIterator iterator, SentenceTransformer transformer)
ParallelTransformerIterator(LabelAwareIterator iterator, SentenceTransformer transformer, boolean allowMultithreading)

Method Summary

Modifier and Type	Method and Description
boolean	hasNext()
Sequence<VocabWord>	next()
void	reset()

Methods inherited from class org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.BasicTransformerIterator

remove

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

buffer

protected BlockingQueue<Sequence<VocabWord>> buffer

stringBuffer

protected BlockingQueue<LabelledDocument> stringBuffer

threads

protected org.deeplearning4j.models.sequencevectors.transformers.impl.iterables.ParallelTransformerIterator.TokenizerThread[] threads

underlyingHas

protected boolean underlyingHas

processing

protected AtomicInteger processing

count

protected static final AtomicInteger count

Constructor Detail

ParallelTransformerIterator

public ParallelTransformerIterator(@NonNull
                                   LabelAwareIterator iterator,
                                   @NonNull
                                   SentenceTransformer transformer)

ParallelTransformerIterator

public ParallelTransformerIterator(@NonNull
                                   LabelAwareIterator iterator,
                                   @NonNull
                                   SentenceTransformer transformer,
                                   boolean allowMultithreading)

Method Detail

reset

public void reset()

Overrides:

reset in class BasicTransformerIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<Sequence<VocabWord>>

Overrides:

hasNext in class BasicTransformerIterator

next

public Sequence<VocabWord> next()

Specified by:

next in interface Iterator<Sequence<VocabWord>>

Overrides:

next in class BasicTransformerIterator