org.stringtemplate.v4.compiler

Class STLexer

java.lang.Object

org.stringtemplate.v4.compiler.STLexer

All Implemented Interfaces:

org.antlr.runtime.TokenSource

public class STLexer
extends Object
implements org.antlr.runtime.TokenSource

This class represents the tokenizer for templates. It operates in two modes: inside and outside of expressions. It implements the TokenSource interface so it can be used with ANTLR parsers. Outside of expressions, we can return these token types: TEXT, INDENT, LDELIM (start of expression), RCURLY (end of subtemplate), and NEWLINE. Inside of an expression, this lexer returns all of the tokens needed by STParser. From the parser's point of view, it can treat a template as a simple stream of elements.

This class defines the token types and communicates these values to STParser.g via STLexer.tokens file (which must remain consistent).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	STLexer.STToken We build STToken tokens instead of relying on CommonToken so we can override STLexer.STToken.toString().

Field Summary

Fields

Modifier and Type	Field and Description
static int	AND
static int	AT
static int	BANG
static int	COLON
static int	COMMA
static int	COMMENT
static int	DOT
static int	ELLIPSIS
static int	ELSE
static int	ELSEIF
static int	ENDIF
static char	EOF
static int	EOF_TYPE
static int	EQUALS
static int	FALSE
static int	ID
static int	IF
static int	INDENT
static int	LBRACK
static int	LCURLY
static int	LDELIM
static int	LPAREN
static int	NEWLINE
static int	OR
static int	PIPE
static int	RBRACK
static int	RCURLY
static int	RDELIM
static int	REGION_END
static int	RPAREN
static int	SEMI
static org.antlr.runtime.Token	SKIP
static int	STRING
int	subtemplateDepth To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates.
static int	SUPER
static int	TEXT
static int	TRUE

Constructor Summary

Constructors

Constructor and Description
STLexer(org.antlr.runtime.CharStream input)
STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken)
STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar)

Method Summary

Methods

Modifier and Type	Method and Description
org.antlr.runtime.Token	_nextToken()
protected void	consume()
void	emit(org.antlr.runtime.Token token)
String	getSourceName()
protected org.antlr.runtime.Token	inside()
static boolean	isIDLetter(char c)
static boolean	isIDStartLetter(char c)
static boolean	isUnicodeLetter(char c)
static boolean	isWS(char c)
void	match(char x) Consume if x is next character on the input stream.
org.antlr.runtime.Token	newToken(int ttype)
org.antlr.runtime.Token	newToken(int ttype, String text)
org.antlr.runtime.Token	newToken(int ttype, String text, int pos)
org.antlr.runtime.Token	newTokenFromPreviousChar(int ttype)
org.antlr.runtime.Token	nextToken()
protected org.antlr.runtime.Token	outside()
static String	str(int c)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EOF

public static final char EOF

See Also:

Constant Field Values

EOF_TYPE

public static final int EOF_TYPE

See Also:

Constant Field Values

SKIP

public static final org.antlr.runtime.Token SKIP

RBRACK

public static final int RBRACK

See Also:

Constant Field Values

LBRACK

public static final int LBRACK

See Also:

Constant Field Values

ELSE

public static final int ELSE

See Also:

Constant Field Values

ELLIPSIS

public static final int ELLIPSIS

See Also:

Constant Field Values

LCURLY

public static final int LCURLY

See Also:

Constant Field Values

BANG

public static final int BANG

See Also:

Constant Field Values

EQUALS

public static final int EQUALS

See Also:

Constant Field Values

TEXT

public static final int TEXT

See Also:

Constant Field Values

ID

public static final int ID

See Also:

Constant Field Values

SEMI

public static final int SEMI

See Also:

Constant Field Values

LPAREN

public static final int LPAREN

See Also:

Constant Field Values

IF

public static final int IF

See Also:

Constant Field Values

ELSEIF

public static final int ELSEIF

See Also:

Constant Field Values

COLON

public static final int COLON

See Also:

Constant Field Values

RPAREN

public static final int RPAREN

See Also:

Constant Field Values

COMMA

public static final int COMMA

See Also:

Constant Field Values

RCURLY

public static final int RCURLY

See Also:

Constant Field Values

ENDIF

public static final int ENDIF

See Also:

Constant Field Values

RDELIM

public static final int RDELIM

See Also:

Constant Field Values

SUPER

public static final int SUPER

See Also:

Constant Field Values

DOT

public static final int DOT

See Also:

Constant Field Values

LDELIM

public static final int LDELIM

See Also:

Constant Field Values

STRING

public static final int STRING

See Also:

Constant Field Values

PIPE

public static final int PIPE

See Also:

Constant Field Values

OR

public static final int OR

See Also:

Constant Field Values

AND

public static final int AND

See Also:

Constant Field Values

INDENT

public static final int INDENT

See Also:

Constant Field Values

NEWLINE

public static final int NEWLINE

See Also:

Constant Field Values

AT

public static final int AT

See Also:

Constant Field Values

REGION_END

public static final int REGION_END

See Also:

Constant Field Values

TRUE

public static final int TRUE

See Also:

Constant Field Values

FALSE

public static final int FALSE

See Also:

Constant Field Values

COMMENT

public static final int COMMENT

See Also:

Constant Field Values

subtemplateDepth

public int subtemplateDepth

To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates. Otherwise, we know whether a '}' and the outermost subtemplate to send this back to outside mode.

Constructor Detail

STLexer

public STLexer(org.antlr.runtime.CharStream input)

STLexer

public STLexer(ErrorManager errMgr,
       org.antlr.runtime.CharStream input,
       org.antlr.runtime.Token templateToken)

STLexer

public STLexer(ErrorManager errMgr,
       org.antlr.runtime.CharStream input,
       org.antlr.runtime.Token templateToken,
       char delimiterStartChar,
       char delimiterStopChar)

Method Detail

nextToken

public org.antlr.runtime.Token nextToken()

Specified by:

nextToken in interface org.antlr.runtime.TokenSource

match

public void match(char x)

Consume if x is next character on the input stream.

Throws:

org.antlr.runtime.NoViableAltException

consume

protected void consume()

emit

public void emit(org.antlr.runtime.Token token)

_nextToken

public org.antlr.runtime.Token _nextToken()

outside

protected org.antlr.runtime.Token outside()

inside

protected org.antlr.runtime.Token inside()

isIDStartLetter

public static boolean isIDStartLetter(char c)

isIDLetter

public static boolean isIDLetter(char c)

isWS

public static boolean isWS(char c)

isUnicodeLetter

public static boolean isUnicodeLetter(char c)

newToken

public org.antlr.runtime.Token newToken(int ttype)

newTokenFromPreviousChar

public org.antlr.runtime.Token newTokenFromPreviousChar(int ttype)

newToken

public org.antlr.runtime.Token newToken(int ttype,
                               String text,
                               int pos)

newToken

public org.antlr.runtime.Token newToken(int ttype,
                               String text)

getSourceName

public String getSourceName()

Specified by:

getSourceName in interface org.antlr.runtime.TokenSource

str

public static String str(int c)