org.languagetool.tokenizers

Interface Tokenizer

All Known Subinterfaces:

CompoundWordTokenizer, SentenceTokenizer

All Known Implementing Classes:

LocalSRXSentenceTokenizer, SimpleSentenceTokenizer, SRXSentenceTokenizer, WordTokenizer

public interface Tokenizer

Interface for classes that tokenize text into smaller units.

Author:

Daniel Naber

Method Summary

Methods

Modifier and Type	Method and Description
List<String>	tokenize(String text)

Method Detail

tokenize

List<String> tokenize(String text)