org.languagetool.tagging

Class BaseTagger

java.lang.Object

org.languagetool.tagging.BaseTagger

All Implemented Interfaces:

Tagger

public abstract class BaseTagger
extends Object
implements Tagger

Base tagger using Morfologik binary dictionaries.

Author:

Marcin Milkowski

Field Summary

Fields

Modifier and Type	Field and Description
protected Locale	conversionLocale
protected WordTagger	wordTagger

Constructor Summary

Constructors

Constructor and Description
BaseTagger(String filename)
BaseTagger(String filename, Locale conversionLocale)
BaseTagger(String filename, Locale locale, boolean tagLowercaseWithUppercase)

Method Summary

Methods

Modifier and Type	Method and Description
protected List<AnalyzedToken>	additionalTags(String word, WordTagger wordTagger) Allows additional tagging in some language-dependent circumstances
protected AnalyzedToken	asAnalyzedToken(String word, morfologik.stemming.WordData wd)
protected List<AnalyzedToken>	asAnalyzedTokenList(String word, List<morfologik.stemming.WordData> wdList)
protected List<AnalyzedToken>	asAnalyzedTokenListForTaggedWords(String word, List<TaggedWord> taggedWords)
AnalyzedTokenReadings	createNullToken(String token, int startPos) Create the AnalyzedToken used for whitespace and other non-words.
AnalyzedToken	createToken(String token, String posTag) Create a token specific to the language of the implementing class.
protected List<AnalyzedToken>	getAnalyzedTokens(String word)
protected morfologik.stemming.Dictionary	getDictionary()
String	getDictionaryPath()
abstract String	getManualAdditionsFileName() Get the filename for manual additions, e.g., /en/added.txt, or null.
protected WordTagger	getWordTagger()
boolean	overwriteWithManualTagger() If true, tags from the binary dictionary (*.dict) will be overwritten by manual tags from the plain text dictionary.
List<AnalyzedTokenReadings>	tag(List<String> sentenceTokens) Returns a list of AnalyzedTokens that assigns each term in the sentence some kind of part-of-speech information (not necessarily just one tag).
protected List<AnalyzedTokenReadings>	tag(String token)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

wordTagger

protected final WordTagger wordTagger

conversionLocale

protected final Locale conversionLocale

Constructor Detail

BaseTagger

public BaseTagger(String filename)

Since:

2.9

BaseTagger

public BaseTagger(String filename,
          Locale conversionLocale)

Since:

2.9

BaseTagger

public BaseTagger(String filename,
          Locale locale,
          boolean tagLowercaseWithUppercase)

Since:

2.9

Method Detail

getManualAdditionsFileName

@Nullable
public abstract String getManualAdditionsFileName()

Get the filename for manual additions, e.g., /en/added.txt, or null.

Since:

2.8

getDictionaryPath

public String getDictionaryPath()

Since:

2.9

overwriteWithManualTagger

public boolean overwriteWithManualTagger()

If true, tags from the binary dictionary (*.dict) will be overwritten by manual tags from the plain text dictionary.

Since:

2.9

getWordTagger

protected WordTagger getWordTagger()

getDictionary

protected morfologik.stemming.Dictionary getDictionary()

tag

public List<AnalyzedTokenReadings> tag(List<String> sentenceTokens)
                                throws IOException

Description copied from interface: Tagger

Returns a list of AnalyzedTokens that assigns each term in the sentence some kind of part-of-speech information (not necessarily just one tag).

Note that this method takes exactly one sentence. Its implementation may implement special cases for the first word of a sentence, which is usually written with an uppercase letter.

Specified by:

tag in interface Tagger

Parameters:

sentenceTokens - the text as returned by a WordTokenizer

Throws:

IOException

tag

protected List<AnalyzedTokenReadings> tag(String token)
                                   throws IOException

Throws:

IOException

getAnalyzedTokens

protected List<AnalyzedToken> getAnalyzedTokens(String word)

asAnalyzedTokenList

protected List<AnalyzedToken> asAnalyzedTokenList(String word,
                                      List<morfologik.stemming.WordData> wdList)

asAnalyzedTokenListForTaggedWords

protected List<AnalyzedToken> asAnalyzedTokenListForTaggedWords(String word,
                                                    List<TaggedWord> taggedWords)

asAnalyzedToken

protected AnalyzedToken asAnalyzedToken(String word,
                            morfologik.stemming.WordData wd)

createNullToken

public final AnalyzedTokenReadings createNullToken(String token,
                                    int startPos)

Description copied from interface: Tagger

Create the AnalyzedToken used for whitespace and other non-words. Use null as the POS tag for this token.

Specified by:

createNullToken in interface Tagger

createToken

public AnalyzedToken createToken(String token,
                        String posTag)

Description copied from interface: Tagger

Create a token specific to the language of the implementing class.

Specified by:

createToken in interface Tagger

additionalTags

@Nullable
protected List<AnalyzedToken> additionalTags(String word,
                                          WordTagger wordTagger)

Allows additional tagging in some language-dependent circumstances

Parameters:

word - The word to tag

Returns:

Returns list of analyzed tokens with additional tags, or null