org.languagetool.tokenizers.br

Class BretonWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.br.BretonWordTokenizer

All Implemented Interfaces:

Tokenizer

public class BretonWordTokenizer
extends WordTokenizer

Author:

Dominique Pelle

Constructor Summary

Constructors

Constructor and Description
BretonWordTokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
List<String>	tokenize(String text) Tokenizes just like WordTokenizer with the exception that "c’h" is not split.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, getTokenizingCharacters, isUrl, joinUrls

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BretonWordTokenizer

public BretonWordTokenizer()

Method Detail

tokenize

public List<String> tokenize(String text)

Tokenizes just like WordTokenizer with the exception that "c’h" is not split. "C’h" is considered as a letter in breton (trigraph) and it occurs in many words. So tokenizer should not split it. Also split things like "n’eo" into 2 tokens only "n’" + "eo".

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - - Text to tokenize

Returns:

List of tokens. Note: a special string ##BR_APOS## is used to replace apostrophes during tokenizing.