Uses of Interface
org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory

Packages that use TokenizerFactory

Package	Description
org.deeplearning4j.bagofwords.vectorizer
org.deeplearning4j.iterator
org.deeplearning4j.models.embeddings.loader
org.deeplearning4j.models.glove
org.deeplearning4j.models.paragraphvectors
org.deeplearning4j.models.sequencevectors.transformers.impl
org.deeplearning4j.models.word2vec
org.deeplearning4j.text.movingwindow
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of TokenizerFactory in org.deeplearning4j.bagofwords.vectorizer

Fields in org.deeplearning4j.bagofwords.vectorizer declared as TokenizerFactory

Modifier and Type	Field and Description
protected TokenizerFactory	BagOfWordsVectorizer.Builder.tokenizerFactory
protected TokenizerFactory	TfidfVectorizer.Builder.tokenizerFactory
protected TokenizerFactory	BaseTextVectorizer.tokenizerFactory
protected TokenizerFactory	Builder.tokenizerFactory Deprecated.

Methods in org.deeplearning4j.bagofwords.vectorizer with parameters of type TokenizerFactory

Modifier and Type	Method and Description
BagOfWordsVectorizer.Builder	BagOfWordsVectorizer.Builder.setTokenizerFactory(TokenizerFactory tokenizerFactory)
TfidfVectorizer.Builder	TfidfVectorizer.Builder.setTokenizerFactory(TokenizerFactory tokenizerFactory)
Builder	Builder.tokenize(TokenizerFactory tokenizerFactory) Deprecated.

Uses of TokenizerFactory in org.deeplearning4j.iterator

Methods in org.deeplearning4j.iterator with parameters of type TokenizerFactory

Modifier and Type	Method and Description
CnnSentenceDataSetIterator.Builder	CnnSentenceDataSetIterator.Builder.tokenizerFactory(TokenizerFactory tokenizerFactory) The TokenizerFactory that should be used.

Uses of TokenizerFactory in org.deeplearning4j.models.embeddings.loader

Methods in org.deeplearning4j.models.embeddings.loader that return TokenizerFactory

Modifier and Type	Method and Description
protected static TokenizerFactory	WordVectorSerializer.getTokenizerFactory(VectorsConfiguration configuration)

Uses of TokenizerFactory in org.deeplearning4j.models.glove

Fields in org.deeplearning4j.models.glove declared as TokenizerFactory

Modifier and Type	Field and Description
protected TokenizerFactory	Glove.Builder.tokenFactory

Methods in org.deeplearning4j.models.glove with parameters of type TokenizerFactory

Modifier and Type	Method and Description
Glove.Builder	Glove.Builder.tokenizerFactory(TokenizerFactory tokenizerFactory) Sets TokenizerFactory to be used for training

Uses of TokenizerFactory in org.deeplearning4j.models.paragraphvectors

Methods in org.deeplearning4j.models.paragraphvectors with parameters of type TokenizerFactory

Modifier and Type	Method and Description
ParagraphVectors.Builder	ParagraphVectors.Builder.tokenizerFactory(TokenizerFactory tokenizerFactory) This method defines TokenizerFactory to be used for strings tokenization during training PLEASE NOTE: If external VocabCache is used, the same TokenizerFactory should be used to keep derived tokens equal.

Constructors in org.deeplearning4j.models.paragraphvectors with parameters of type TokenizerFactory

Constructor and Description
BlindInferenceCallable(VocabCache<VocabWord> vocabCache, TokenizerFactory tokenizerFactory, String document)
BlindInferenceCallable(VocabCache<VocabWord> vocabCache, TokenizerFactory tokenizerFactory, String document, AtomicLong flag)
InferenceCallable(VocabCache<VocabWord> vocabCache, TokenizerFactory tokenizerFactory, LabelledDocument document)
InferenceCallable(VocabCache<VocabWord> vocabCache, TokenizerFactory tokenizerFactory, LabelledDocument document, AtomicLong flag)

Uses of TokenizerFactory in org.deeplearning4j.models.sequencevectors.transformers.impl

Fields in org.deeplearning4j.models.sequencevectors.transformers.impl declared as TokenizerFactory

Modifier and Type	Field and Description
protected TokenizerFactory	SentenceTransformer.tokenizerFactory
protected TokenizerFactory	SentenceTransformer.Builder.tokenizerFactory

Methods in org.deeplearning4j.models.sequencevectors.transformers.impl with parameters of type TokenizerFactory

Modifier and Type	Method and Description
SentenceTransformer.Builder	SentenceTransformer.Builder.tokenizerFactory(TokenizerFactory tokenizerFactory)

Uses of TokenizerFactory in org.deeplearning4j.models.word2vec

Fields in org.deeplearning4j.models.word2vec declared as TokenizerFactory

Modifier and Type	Field and Description
protected TokenizerFactory	Word2Vec.tokenizerFactory
protected TokenizerFactory	Word2Vec.Builder.tokenizerFactory

Methods in org.deeplearning4j.models.word2vec with parameters of type TokenizerFactory

Modifier and Type	Method and Description
void	Word2Vec.setTokenizerFactory(TokenizerFactory tokenizerFactory) This method defines TokenizerFactory instance to be using during model building
Word2Vec.Builder	Word2Vec.Builder.tokenizerFactory(TokenizerFactory tokenizerFactory) This method defines TokenizerFactory to be used for strings tokenization during training PLEASE NOTE: If external VocabCache is used, the same TokenizerFactory should be used to keep derived tokens equal.

Uses of TokenizerFactory in org.deeplearning4j.text.movingwindow

Methods in org.deeplearning4j.text.movingwindow with parameters of type TokenizerFactory

Modifier and Type	Method and Description
static org.nd4j.linalg.primitives.Pair<String,MultiDimensionalMap<Integer,Integer,String>>	ContextLabelRetriever.stringWithLabels(String sentence, TokenizerFactory tokenizerFactory) Returns a stripped sentence with the indices of words with certain kinds of labels.
static List<Window>	Windows.windows(InputStream words, TokenizerFactory tokenizerFactory, int windowSize) Constructs a list of window of size windowSize.
static List<Window>	Windows.windows(String words, TokenizerFactory tokenizerFactory) Constructs a list of window of size windowSize.
static List<Window>	Windows.windows(String words, TokenizerFactory tokenizerFactory, int windowSize, WordVectors vectors) Constructs a list of window of size windowSize.

Uses of TokenizerFactory in org.deeplearning4j.text.tokenization.tokenizerfactory

Classes in org.deeplearning4j.text.tokenization.tokenizerfactory that implement TokenizerFactory

Modifier and Type	Class and Description
class	DefaultTokenizerFactory Default tokenizer based on string tokenizer or stream tokenizer
class	NGramTokenizerFactory

Constructors in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenizerFactory

Constructor and Description
NGramTokenizerFactory(TokenizerFactory tokenizerFactory, Integer minN, Integer maxN)