net.sf.okapi.filters.abstractmarkup

Class AbstractMarkupFilter

java.lang.Object

net.sf.okapi.common.filters.AbstractFilter

net.sf.okapi.filters.abstractmarkup.AbstractMarkupFilter

All Implemented Interfaces:

AutoCloseable, Iterator<Event>, IFilter

Direct Known Subclasses:

ContentFilter, HtmlFilter, XmlStreamFilter

public abstract class AbstractMarkupFilter
extends AbstractFilter

Abstract class useful for creating an IFilter around the Jericho parser. Jericho can parse non-wellformed HTML, XHTML, XML and various server side scripting languages such as PHP, Mason, Perl (all configurable from Jericho). AbstractMarkupFilter takes care of the parser initialization and provides default handlers for each token type returned by the parser.

Handling of translatable text, inline tags, translatable and read-only attributes are configurable through a user defined YAML file. See the Okapi HtmlFilter with defaultConfiguration.yml and OpenXml filters for examples.

Field Summary

Fields inherited from interface net.sf.okapi.common.filters.IFilter

SUB_FILTER

Constructor Summary

Constructors

Constructor and Description
AbstractMarkupFilter() Default constructor for AbstractMarkupFilter using default AbstractMarkupEventBuilder
AbstractMarkupFilter(AbstractMarkupEventBuilder eventBuilder) Default constructor for AbstractMarkupFilter using specified AbstractMarkupEventBuilder

Method Summary

Modifier and Type	Method and Description
protected void	addCodeToCurrentTextUnit(net.htmlparser.jericho.Tag tag) Add an Code to the current TextUnit.
protected void	addCodeToCurrentTextUnit(net.htmlparser.jericho.Tag tag, boolean endCodeNow) Add an Code to the current TextUnit.
protected void	addFilterEvent(Event event)
protected void	addToDocumentPart(String part)
protected void	addToTextUnit(Code code)
protected void	addToTextUnit(Code code, boolean endCodeNow)
protected void	addToTextUnit(Code code, boolean endCodeNow, List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)
protected void	addToTextUnit(Code code, List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)
protected void	addToTextUnit(String text)
protected void	appendToFirstSkeletonPart(String text)
protected boolean	canStartNewTextUnit()
void	close() Close the filter and all used resources.
protected AbstractMarkupEventBuilder	createEventBuilder() Delayed initialization of the EventBuilder.
protected PropertyTextUnitPlaceholder	createPropertyTextUnitPlaceholder(PropertyTextUnitPlaceholder.PlaceholderAccessType type, String name, String value, net.htmlparser.jericho.Tag tag, net.htmlparser.jericho.Attribute attribute) Create a PropertyTextUnitPlaceholder given the supplied type, name and Jericho Tag and Attribute.
protected List<PropertyTextUnitPlaceholder>	createPropertyTextUnitPlaceholders(net.htmlparser.jericho.StartTag startTag) For the given Jericho StartTag parse out all the actionable attributes and and store them as PropertyTextUnitPlaceholder.
protected String	detectEncoding(RawDocument input)
protected TextFragment.TagType	determineTagType(net.htmlparser.jericho.Tag tag) Filter specific method for determining TextFragment.TagType
protected void	endDocumentPart()
protected void	endFilter() End the current filter processing and send the Ending Event
protected void	endGroup(GenericSkeleton endMarker)
protected void	endTextUnit(GenericSkeleton endMarker)
StringBuilder	getBufferedWhiteSpace()
protected abstract TaggedFilterConfiguration	getConfig() Get the current TaggedFilterConfiguration.
protected String	getCurrentDocName()
protected long	getDocumentPartId()
AbstractMarkupEventBuilder	getEventBuilder()
protected long	getGroupIdSequence()
protected net.htmlparser.jericho.Source	getParsedHeader(InputStream inputStream)
protected ExtractionRuleState	getRuleState()
protected long	getTextUnitId()
protected void	handleCdataSection(net.htmlparser.jericho.Tag tag) Handle CDATA sections.
protected void	handleCharacterEntity(net.htmlparser.jericho.CharacterEntityReference entity) Handle all numeric entities.
protected void	handleComment(net.htmlparser.jericho.Tag tag) Handle comments.
protected void	handleDocTypeDeclaration(net.htmlparser.jericho.Tag tag) Handle the XML doc type declaration (DTD).
protected void	handleDocumentPart(net.htmlparser.jericho.Tag tag) Handle anything else not classified by Jericho.
protected void	handleEndTag(net.htmlparser.jericho.EndTag endTag) Handle end tags, including empty tags.
protected void	handleMarkupDeclaration(net.htmlparser.jericho.Tag tag) Handle an XML markup declaration.
protected void	handleNumericEntity(net.htmlparser.jericho.NumericCharacterReference entity) Handle all Character entities.
protected void	handleProcessingInstruction(net.htmlparser.jericho.Tag tag) Handle processing instructions.
protected void	handleServerCommon(net.htmlparser.jericho.Tag tag) Handle any recognized server tags (i.e., PHP, Mason etc.)
protected void	handleServerCommonEscaped(net.htmlparser.jericho.Tag tag) Handle any recognized escaped server tags.
protected void	handleStartTag(net.htmlparser.jericho.StartTag startTag) Handle start tags.
protected void	handleText(CharSequence text) Handle all text (PCDATA).
protected void	handleXmlDeclaration(net.htmlparser.jericho.Tag tag) Handle an XML declaration.
boolean	hasNext() Indicates if there is an event to process.
protected boolean	isBOM() Does the input have a BOM?
protected boolean	isDocumentEncoding() Does this document have a document encoding specified?
protected boolean	isInsideTextRun()
protected boolean	isPreserveWhitespace()
protected boolean	isUtf8Bom() Does the input have a UTF-8 Byte Order Mark?
protected boolean	isUtf8Encoding() Is the input encoded as UTF-8?
protected boolean	isWhiteSpace(CharSequence text)
Event	next() Queue up Jericho tokens until we can build an Okapi Event and return it.
protected abstract String	normalizeAttributeName(String attrName, String attrValue, net.htmlparser.jericho.Tag tag) Some attributes names are converted to Okapi standards such as HTML charset to "encoding" and lang to "language"
void	open(RawDocument input) Start a new IFilter using the supplied RawDocument.
void	open(RawDocument input, boolean generateSkeleton) Start a new IFilter using the supplied RawDocument.
protected Event	peekTempEvent()
protected Event	popTempEvent()
protected void	postProcessTextUnit(ITextUnit textUnit) Do any required post-processing on the TextUnit before the Event leaves the IFilter.
protected void	preProcess(net.htmlparser.jericho.Segment segment) Do any handling needed before the current Segment is processed.
protected void	setCurrentDocName(String currentDocName)
protected void	setDocumentPartId(long id)
protected void	setGroupIdSequence(long id)
void	setMimeType(String mimeType) Sets the input document mime type.
protected void	setPreserveWhitespace(boolean preserveWhitespace)
protected void	setTextUnitId(long id)
protected void	setTextUnitMimeType(String mimeType)
protected void	setTextUnitName(String name)
protected void	setTextUnitPreserveWhitespace(boolean preserveWhitespace)
protected void	setTextUnitTranslatable(boolean translatable)
protected void	setTextUnitType(String type)
protected void	startDocumentPart(String part, String name, List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)
protected void	startFilter() Initialize the filter for every input and send the StartDocument Event
protected void	startGroup(GenericSkeleton startMarker, String commonTagType)
protected void	startGroup(GenericSkeleton startMarker, String commonTagType, LocaleId locale, List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)
protected void	startTextUnit()
protected void	startTextUnit(GenericSkeleton startMarker)
protected void	startTextUnit(GenericSkeleton startMarker, List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)
protected void	startTextUnit(String text)
protected TaggedFilterConfiguration.RULE_TYPE	updateEndTagRuleState(net.htmlparser.jericho.EndTag endTag)
protected void	updateStartTagRuleState(String tag, TaggedFilterConfiguration.RULE_TYPE ruleType, String idValue)

Methods inherited from class net.sf.okapi.common.filters.AbstractFilter

addConfiguration, addConfiguration, addConfiguration, addConfigurations, cancel, createEndFilterEvent, createFilterWriter, createSkeletonWriter, createStartFilterEvent, findConfiguration, getConfiguration, getConfigurations, getDisplayName, getDocumentId, getDocumentName, getEncoderManager, getEncoding, getFilterConfigurationMapper, getFilterWriter, getMimeType, getName, getNewlineType, getParameters, getParameters, getParametersClassName, getParentId, getSrcLoc, getTrgLoc, isCanceled, isGenerateSkeleton, isMultilingual, removeConfiguration, setDisplayName, setDocumentName, setEncoding, setFilterConfigurationMapper, setFilterWriter, setGenerateSkeleton, setMultilingual, setName, setNewlineType, setOptions, setParameters, setParentId, setSrcLoc, setTrgLoc

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.sf.okapi.common.filters.IFilter

stream

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

AbstractMarkupFilter

public AbstractMarkupFilter()

Default constructor for AbstractMarkupFilter using default AbstractMarkupEventBuilder

AbstractMarkupFilter

public AbstractMarkupFilter(AbstractMarkupEventBuilder eventBuilder)

Default constructor for AbstractMarkupFilter using specified AbstractMarkupEventBuilder

Method Detail

getConfig

protected abstract TaggedFilterConfiguration getConfig()

Get the current TaggedFilterConfiguration. A TaggedFilterConfiguration is the result of reading in a YAML configuration file and converting it into Java Objects.

Returns:

a TaggedFilterConfiguration

close

public void close()

Close the filter and all used resources.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface IFilter

Overrides:

close in class AbstractFilter

getParsedHeader

protected net.htmlparser.jericho.Source getParsedHeader(InputStream inputStream)

detectEncoding

protected String detectEncoding(RawDocument input)

open

public void open(RawDocument input)

Start a new IFilter using the supplied RawDocument.

Parameters:

input - - input to the IFilter (can be a CharSequence, URI or InputStream)

open

public void open(RawDocument input,
                 boolean generateSkeleton)

Start a new IFilter using the supplied RawDocument.

Specified by:

open in interface IFilter

Overrides:

open in class AbstractFilter

Parameters:

input - - input to the IFilter (can be a CharSequence, URI or InputStream)

generateSkeleton - - true if the IFilter should store non-translatble blocks (aka skeleton), false otherwise.

Throws:

OkapiBadFilterInputException

OkapiIOException

hasNext

public boolean hasNext()

Description copied from interface: IFilter

Indicates if there is an event to process.

Implementer Note: The caller must be able to call this method several times without changing state.

Returns:

True if there is at least one event to process, false if not.

next

public Event next()

Queue up Jericho tokens until we can build an Okapi Event and return it.

Returns:

The next event available or null if there are no events.

createEventBuilder

protected AbstractMarkupEventBuilder createEventBuilder()

Delayed initialization of the EventBuilder. This will be called when the filter is initialized if an EventBuilder was not previously passed to the constructor.

Returns:

startFilter

protected void startFilter()

Initialize the filter for every input and send the StartDocument Event

endFilter

protected void endFilter()

End the current filter processing and send the Ending Event

preProcess

protected void preProcess(net.htmlparser.jericho.Segment segment)

Do any handling needed before the current Segment is processed. Default is to do nothing.

Parameters:

segment -

postProcessTextUnit

protected void postProcessTextUnit(ITextUnit textUnit)

Do any required post-processing on the TextUnit before the Event leaves the IFilter. Default implementation leaves Event unchanged. Override this method if you need to do format specific handing such as collapsing whitespace.

handleServerCommonEscaped

protected void handleServerCommonEscaped(net.htmlparser.jericho.Tag tag)

Handle any recognized escaped server tags.

Parameters:

tag -

handleServerCommon

protected void handleServerCommon(net.htmlparser.jericho.Tag tag)

Handle any recognized server tags (i.e., PHP, Mason etc.)

Parameters:

tag -

handleMarkupDeclaration

protected void handleMarkupDeclaration(net.htmlparser.jericho.Tag tag)

Handle an XML markup declaration.

Parameters:

tag -

handleXmlDeclaration

protected void handleXmlDeclaration(net.htmlparser.jericho.Tag tag)

Handle an XML declaration.

Parameters:

tag -

handleDocTypeDeclaration

protected void handleDocTypeDeclaration(net.htmlparser.jericho.Tag tag)

Handle the XML doc type declaration (DTD).

Parameters:

tag -

handleProcessingInstruction

protected void handleProcessingInstruction(net.htmlparser.jericho.Tag tag)

Handle processing instructions.

Parameters:

tag -

handleComment

protected void handleComment(net.htmlparser.jericho.Tag tag)

Handle comments.

Parameters:

tag -

handleCdataSection

protected void handleCdataSection(net.htmlparser.jericho.Tag tag)

Handle CDATA sections.

Parameters:

tag -

handleText

protected void handleText(CharSequence text)

Handle all text (PCDATA).

Parameters:

text -

isWhiteSpace

protected boolean isWhiteSpace(CharSequence text)

handleNumericEntity

protected void handleNumericEntity(net.htmlparser.jericho.NumericCharacterReference entity)

Handle all Character entities. Default implementation converts entity to Unicode character.

Parameters:

entity - - the character entity

handleCharacterEntity

protected void handleCharacterEntity(net.htmlparser.jericho.CharacterEntityReference entity)

Handle all numeric entities. Default implementation converts entity to Unicode character.

Parameters:

entity - - the numeric entity

handleStartTag

protected void handleStartTag(net.htmlparser.jericho.StartTag startTag)

Handle start tags.

Parameters:

startTag -

updateStartTagRuleState

protected void updateStartTagRuleState(String tag,
                                       TaggedFilterConfiguration.RULE_TYPE ruleType,
                                       String idValue)

updateEndTagRuleState

protected TaggedFilterConfiguration.RULE_TYPE updateEndTagRuleState(net.htmlparser.jericho.EndTag endTag)

handleEndTag

protected void handleEndTag(net.htmlparser.jericho.EndTag endTag)

Handle end tags, including empty tags.

Parameters:

endTag -

handleDocumentPart

protected void handleDocumentPart(net.htmlparser.jericho.Tag tag)

Handle anything else not classified by Jericho.

Parameters:

tag -

normalizeAttributeName

protected abstract String normalizeAttributeName(String attrName,
                                                 String attrValue,
                                                 net.htmlparser.jericho.Tag tag)

Some attributes names are converted to Okapi standards such as HTML charset to "encoding" and lang to "language"

Parameters:

attrName - - the attribute name

attrValue - - the attribute value

tag - - the Jericho Tag that contains the attribute

Returns:

the attribute name after it as passe through the normalization rules

addCodeToCurrentTextUnit

protected void addCodeToCurrentTextUnit(net.htmlparser.jericho.Tag tag)

Add an Code to the current TextUnit. Throws an exception if there is no current TextUnit.

Parameters:

tag - - the Jericho Tag that is converted to a Okpai Code

determineTagType

protected TextFragment.TagType determineTagType(net.htmlparser.jericho.Tag tag)

Filter specific method for determining TextFragment.TagType

Parameters:

tag - Jericho Tag start or end tag

Returns:

PLACEHOLDER, OPEN, CLOSED TextFragment.TagType

addCodeToCurrentTextUnit

protected void addCodeToCurrentTextUnit(net.htmlparser.jericho.Tag tag,
                                        boolean endCodeNow)

Add an Code to the current TextUnit. Throws an exception if there is no current TextUnit.

Parameters:

tag - - the Jericho Tag that is converted to a Okpai Code

endCodeNow - - do we end the code now or delay so we can add more content to the code?

createPropertyTextUnitPlaceholders

protected List<PropertyTextUnitPlaceholder> createPropertyTextUnitPlaceholders(net.htmlparser.jericho.StartTag startTag)

For the given Jericho StartTag parse out all the actionable attributes and and store them as PropertyTextUnitPlaceholder. PropertyTextUnitPlaceholder.PlaceholderAccessType are set based on the filter configuration for each attribute. for the attribute name and value.

Parameters:

startTag - - Jericho StartTag

Returns:

all actionable (translatable, writable or read-only) attributes found in the StartTag

createPropertyTextUnitPlaceholder

protected PropertyTextUnitPlaceholder createPropertyTextUnitPlaceholder(PropertyTextUnitPlaceholder.PlaceholderAccessType type,
                                                                        String name,
                                                                        String value,
                                                                        net.htmlparser.jericho.Tag tag,
                                                                        net.htmlparser.jericho.Attribute attribute)

Create a PropertyTextUnitPlaceholder given the supplied type, name and Jericho Tag and Attribute.

Parameters:

type - - PropertyTextUnitPlaceholder.PlaceholderAccessType is one of TRANSLATABLE, READ_ONLY_PROPERTY, WRITABLE_PROPERTY

name - - attribute name

value - - attribute value

tag - - Jericho Tag which contains the attribute

attribute - - attribute as a Jericho Attribute

Returns:

a PropertyTextUnitPlaceholder representing the attribute

isUtf8Encoding

protected boolean isUtf8Encoding()

Is the input encoded as UTF-8?

Overrides:

isUtf8Encoding in class AbstractFilter

Returns:

true if the document is in utf8 encoding.

isUtf8Bom

protected boolean isUtf8Bom()

Does the input have a UTF-8 Byte Order Mark?

Overrides:

isUtf8Bom in class AbstractFilter

Returns:

true if the document has a utf-8 byte order mark.

isBOM

protected boolean isBOM()

Does the input have a BOM?

Returns:

true if the document has a BOM.

isDocumentEncoding

protected boolean isDocumentEncoding()

Does this document have a document encoding specified?

Returns:

true if has meta tag with encoding, false otherwise

isPreserveWhitespace

protected boolean isPreserveWhitespace()

Returns:

the preserveWhitespace boolean.

setPreserveWhitespace

protected void setPreserveWhitespace(boolean preserveWhitespace)

setTextUnitPreserveWhitespace

protected void setTextUnitPreserveWhitespace(boolean preserveWhitespace)

addToDocumentPart

protected void addToDocumentPart(String part)

addToTextUnit

protected void addToTextUnit(String text)

startTextUnit

protected void startTextUnit(String text)

setTextUnitName

protected void setTextUnitName(String name)

setTextUnitType

protected void setTextUnitType(String type)

setTextUnitTranslatable

protected void setTextUnitTranslatable(boolean translatable)

setCurrentDocName

protected void setCurrentDocName(String currentDocName)

getCurrentDocName

protected String getCurrentDocName()

canStartNewTextUnit

protected boolean canStartNewTextUnit()

isInsideTextRun

protected boolean isInsideTextRun()

addToTextUnit

protected void addToTextUnit(Code code,
                             boolean endCodeNow)

addToTextUnit

protected void addToTextUnit(Code code)

addToTextUnit

protected void addToTextUnit(Code code,
                             boolean endCodeNow,
                             List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)

addToTextUnit

protected void addToTextUnit(Code code,
                             List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)

endDocumentPart

protected void endDocumentPart()

startDocumentPart

protected void startDocumentPart(String part,
                                 String name,
                                 List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)

startGroup

protected void startGroup(GenericSkeleton startMarker,
                          String commonTagType)

startGroup

protected void startGroup(GenericSkeleton startMarker,
                          String commonTagType,
                          LocaleId locale,
                          List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)

startTextUnit

protected void startTextUnit(GenericSkeleton startMarker)

startTextUnit

protected void startTextUnit(GenericSkeleton startMarker,
                             List<PropertyTextUnitPlaceholder> propertyTextUnitPlaceholders)

endTextUnit

protected void endTextUnit(GenericSkeleton endMarker)

endGroup

protected void endGroup(GenericSkeleton endMarker)

startTextUnit

protected void startTextUnit()

getTextUnitId

protected long getTextUnitId()

setTextUnitId

protected void setTextUnitId(long id)

getGroupIdSequence

protected long getGroupIdSequence()

setGroupIdSequence

protected void setGroupIdSequence(long id)

setTextUnitMimeType

protected void setTextUnitMimeType(String mimeType)

getDocumentPartId

protected long getDocumentPartId()

setDocumentPartId

protected void setDocumentPartId(long id)

appendToFirstSkeletonPart

protected void appendToFirstSkeletonPart(String text)

addFilterEvent

protected void addFilterEvent(Event event)

popTempEvent

protected Event popTempEvent()

peekTempEvent

protected Event peekTempEvent()

getRuleState

protected ExtractionRuleState getRuleState()

getEventBuilder

public AbstractMarkupEventBuilder getEventBuilder()

Returns:

the eventBuilder

setMimeType

public void setMimeType(String mimeType)

Sets the input document mime type.

Overrides:

setMimeType in class AbstractFilter

Parameters:

mimeType - the new mime type

getBufferedWhiteSpace

public StringBuilder getBufferedWhiteSpace()