com.optimaize.langdetect.profiles

Interface LanguageProfile

All Known Implementing Classes:

LanguageProfileImpl

public interface LanguageProfile

A language profile knows the locale (language), and contains the n-grams and some statistics.

It is built from a training text that should be fairly large and clean.

It contains the n-grams from the training text in the desired gram sizes (eg 2 and 3-grams), with possible text filters applied for cleaning. Also, rarely occurring n-grams may have been cut to reduce the noise and index size. Use a LanguageProfileBuilder.

The profile may be created at runtime on-the-fly, or it may be loaded from a previously generated text file (see OldLangProfileConverter).

Author:

Fabian Kessler

Method Summary

Methods

Modifier and Type	Method and Description
int	getFrequency(String gram)
List<Integer>	getGramLengths() Tells what the n in n-grams are used here.
LdLocale	getLocale()
long	getMaxGramCount(int gramLength) Tells how often the n-gram with the highest amount of occurrences used in this profile occurred.
long	getMinGramCount(int gramLength) Tells how often the n-gram with the lowest amount of occurrences used in this profile occurred.
long	getNumGramOccurrences(int gramLength) Tells how often all n-grams of a certain length occurred, combined.
int	getNumGrams() Tells how many n-grams there are for all n-gram sizes combined.
int	getNumGrams(int gramLength) Tells how many different n-grams there are for a certain n-gram size.
Iterable<Map.Entry<String,Integer>>	iterateGrams() Iterates all ngram strings with frequency.
Iterable<Map.Entry<String,Integer>>	iterateGrams(int gramLength) Iterates all gramLength-gram strings with frequency.

Method Detail

getLocale

@NotNull
LdLocale getLocale()

getGramLengths

@NotNull
List<Integer> getGramLengths()

Tells what the n in n-grams are used here. Example: [1,2,3]

Returns:

Sorted from smaller to larger.

getFrequency

int getFrequency(String gram)

Parameters:

gram - for example "a" or "foo".

Returns:

0-n, also zero if this profile does not use n-grams of that length (for example if no 4-grams are made).

getNumGrams

int getNumGrams(int gramLength)

Tells how many different n-grams there are for a certain n-gram size. For example the English language has about 57 different 1-grams, whereas Chinese in Hani has thousands.

Parameters:

gramLength - 1-n

Returns:

0-n, returns zero if no such n-grams were made (for example if no 4-grams were made), or if all the training text did not contain such long words.

getNumGrams

int getNumGrams()

Tells how many n-grams there are for all n-gram sizes combined.

Returns:

0-n (0 only on an empty profile...)

getNumGramOccurrences

long getNumGramOccurrences(int gramLength)

Tells how often all n-grams of a certain length occurred, combined. This returns a much larger number than getNumGrams(int).

Parameters:

gramLength - 1-n

Returns:

0-n, returns zero if no such n-grams were made (for example if no 4-grams were made), or if all the training text did not contain such long words.

getMinGramCount

long getMinGramCount(int gramLength)

Tells how often the n-gram with the lowest amount of occurrences used in this profile occurred. Most likely there were n-grams with less (unless the returned number is 1), but they were eliminated in order to keep the profile reasonably small. This is the opposite of getMaxGramCount().

Parameters:

gramLength - 1-n

Returns:

0-n, returns zero if no such n-grams were made or existed.

getMaxGramCount

long getMaxGramCount(int gramLength)

Tells how often the n-gram with the highest amount of occurrences used in this profile occurred. This is the opposite of getMinGramCount().

Parameters:

gramLength - 1-n

Returns:

0-n, returns zero if no such n-grams were made or existed.

iterateGrams

@NotNull
Iterable<Map.Entry<String,Integer>> iterateGrams()

Iterates all ngram strings with frequency.

iterateGrams

@NotNull
Iterable<Map.Entry<String,Integer>> iterateGrams(int gramLength)

Iterates all gramLength-gram strings with frequency.