All Packages  Class Hierarchy  This Package  Previous  Next  Index

Class epp.Token

java.lang.Object
   |
   +----epp.Token

public abstract class Token

extends Object

The Token class itself is an abstract class. There are two subclasses of Token, which are LiteralToken and Symbol. All literal tokens (integer constans, string constants, etc.) are expressed with LiteralToken. All non-literal tokens (keywords such as class, if, while, symbols such as +, -, ==, identifiers such as variable names) are expressed with Symbol.

Plug-ins may extend the lexical analyzer but cannot create a new subclass of class Token. In order to introduce a new token, you must express it with either LiteralToken or Symbol.

The lexical analyzer does not make any distinction of "keywords." That is, keywords such as if and class, variable and method names such as x and y, and operators such as + and - are all expressed with Symbol.

Literals are expressed with class LiteralToken. When you convert this to a string using toString(), the resulting string would be comprised of a literal tag that specifies the type of literal, and a string that specifies the content of the literal. For example, an integer literal 123 would be converted to (int "123"), and a string literal would be converted to (string "abc").

For example, if you lexically analyze the following program,

	if (x == 0) System.out.println("Hello!");

	if
	(
	x
	==
	(int "0")
	)
	System
	.
	out
	.
	println
	(
	(string "Hello!")
	)
	; 

The literal tag and content of the literal can be retrieved using the methods literalTag() and literalContents() respectively. These methods are provided in class Token and can be called without casting to class LiteralToken. For example, if a certain token was a string literal, a program that would print its contents would be like the following.

	Token token = ...;
	if (token.isLiteralToken() && token.literalTag() == :string){
		System.out.println(token.literalContents());
	}

Because EPP is a preprocessor designed to extend a language, the lexical analyzer is written with utmost care in order to reduce the depenency on the language specification as much as possible. That is why the analyzer does not make any distinction of keywords and identifiers. Furthermore, the format of literals are retained as much as possible so that they do not change too much from how they are written in the program. For example, 003.000 would be expressed as (double "003.000") and would not become (double "3.0").

See Also:

Lex, Symbol, LiteralToken

Token()

isLiteralToken()

Returns true if this token is a LiteralToken and false otherwise.

isSymbol()

Returns true if this token is a Symbol and false otherwise.

literalContents()

Returns literalContents if this token is a LiteralToken.

literalTag()

Returns literalTag if this token is a LiteralToken.

print()

Prints this token to Opts.out.

 public Token()

 public boolean isLiteralToken()

Returns true if this token is a LiteralToken and false otherwise.

 public boolean isSymbol()

Returns true if this token is a Symbol and false otherwise.

 public Symbol literalTag()

Returns literalTag if this token is a LiteralToken. If this token is not a LiteralToken a fatal error results.

See Also:

LiteralToken

 public String literalContents()

Returns literalContents if this token is a LiteralToken. If this token is not a LiteralToken a fatal error results.

See Also:

LiteralToken

 public Token print()

Prints this token to Opts.out. This is a method for debugging. Returns itself as a value. Usage example:

       if(lookahead().print() == :"+") ...

All Packages  Class Hierarchy  This Package  Previous  Next  Index