Verwendungen von Schnittstelle
opennlp.tools.tokenize.Tokenizer

Packages, die Tokenizer verwenden

Package	Beschreibung
opennlp.tools.cmdline.parser
opennlp.tools.formats.brat	Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
opennlp.tools.formats.muc	Experimental package related to the MUC corpus format.
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.util.featuregen	This package contains classes for generating sequence features.

Verwendungen von Tokenizer in opennlp.tools.cmdline.parser

Methoden in opennlp.tools.cmdline.parser mit Parametern vom Typ Tokenizer

Modifizierer und Typ	Methode	Beschreibung
static Parse[]	ParserTool.parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses)

Verwendungen von Tokenizer in opennlp.tools.formats.brat

Konstruktoren in opennlp.tools.formats.brat mit Parametern vom Typ Tokenizer

Modifizierer	Konstruktor	Beschreibung
	BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)
	BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)
	BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)	Creates a new BratNameSampleStream.
	BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)	Creates a new BratNameSampleStream.

Verwendungen von Tokenizer in opennlp.tools.formats.muc

Konstruktoren in opennlp.tools.formats.muc mit Parametern vom Typ Tokenizer

Modifizierer	Konstruktor	Beschreibung
	MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)	Initializes a MucNameContentHandler.
protected	MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)	Initializes a MucNameSampleStream.

Verwendungen von Tokenizer in opennlp.tools.tokenize

Klassen in opennlp.tools.tokenize, die Tokenizer implementieren

Modifizierer und Typ	Klasse	Beschreibung
class	SimpleTokenizer	A basic Tokenizer implementation which performs tokenization using character classes.
class	ThreadSafeTokenizerME	A thread-safe version of TokenizerME.
class	TokenizerME	A Tokenizer for converting raw text into separated tokens.
class	WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
class	WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.

Konstruktoren in opennlp.tools.tokenize mit Parametern vom Typ Tokenizer

Modifizierer	Konstruktor	Beschreibung
	TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)	Initializes an instance to evaluate a Tokenizer.
	TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)	Initializes a instance.

Verwendungen von Tokenizer in opennlp.tools.util.featuregen

Konstruktoren in opennlp.tools.util.featuregen mit Parametern vom Typ Tokenizer

Modifizierer	Konstruktor	Beschreibung
	TokenPatternFeatureGenerator(Tokenizer supportTokenizer)	Initializes a TokenPatternFeatureGenerator instance.