smile.nlp.tokenizer

Class BreakIteratorTokenizer

java.lang.Object

smile.nlp.tokenizer.BreakIteratorTokenizer

All Implemented Interfaces:

Tokenizer

public class BreakIteratorTokenizer
extends java.lang.Object
implements Tokenizer

A word tokenizer based on the java.text.BreakIterator, which supports multiple natural languages (selected by locale setting).

Constructor Summary

Constructors

Constructor and Description
BreakIteratorTokenizer() Constructor for the default locale.
BreakIteratorTokenizer(java.util.Locale locale) Constructor for the given locale.

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	split(java.lang.String text) Splits the string into a list of tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BreakIteratorTokenizer

public BreakIteratorTokenizer()

Constructor for the default locale.

BreakIteratorTokenizer

public BreakIteratorTokenizer(java.util.Locale locale)

Constructor for the given locale.

Method Detail

split

public java.lang.String[] split(java.lang.String text)

Description copied from interface: Tokenizer

Splits the string into a list of tokens.

Specified by:

split in interface Tokenizer