org.erights.e.elang.syntax
Class ELexer

java.lang.Object
  |
  +--org.quasiliteral.syntax.BaseLexer
        |
        +--org.erights.e.elang.syntax.ELexer

All Implemented Interfaces:

LexerFace, Marker, PassByProxy

public class ELexer

extends BaseLexer

Safe: Breaks textually input into a stream of tokens according to the E language's defined grammar.

Author:

Mark S. Miller

Field Summary

static AstroBuilder	FOR_ASTS Enabled:
static AstroBuilder	FOR_TERMS Enabled:

Fields inherited from class org.quasiliteral.syntax.BaseLexer

myBuilder, myChar, myContinueCount, myDelayedNextChar, myIndenter, myLData, myLTwine, myNoTabsFlag, myOptStartPos, myOptStartText, myPos, myQuasiFlag

Fields inherited from interface org.quasiliteral.syntax.LexerFace

EOFCHAR, EOFTOK

Fields inherited from interface org.erights.e.elib.serial.PassByProxy

HONORARY, HONORED_NAMES

Constructor Summary

ELexer(LineFeeder optLineFeeder, boolean partialFlag, boolean noTabsFlag)
Enabled:

ELexer(LineFeeder optLineFeeder, boolean partialFlag, boolean noTabsFlag, AstroBuilder builder)
Enabled:

Method Summary

private Astro	continuer(Astro result, int continueCount) Separated out for use by '>'
protected Astro	getNextToken()
private Astro	identifier() Called with myChar as the first character of the identifier.
static boolean	isIdentifierOrKeyword(String str) Enabled: A legal E identifier or keyword is a string whose first character isIdentifierStart, the rest of whose characters are isIdentifierPart.
static boolean	isIdentifierPart(char ch) Enabled: A non-first character of an E identifier may be anything accepted as a non-first character of a Java identifier except '$'.
static boolean	isIdentifierStart(char ch) Enabled: The first character of an E identifier may be anything accepted as the first character of a Java identifier except '$'.
protected boolean	isRestBlank(int start) Consider '#' and '//' comments to be blank as well.
static void	main(String[] args) Enabled: Just for testing.
static ELexer	make(Twine sourceCode, boolean quasiFlag, boolean noTabsFlag) Enabled: @param sourceCode The source code itself
Astro	nextToken() Enabled: @return :Leaf (see AstroBuilder
private short	optKeywordType(String name) If 'name' is a keyword, return it's token tag code, else -1.
private Astro	optUri() Eat a URI or a URIStart.
private Astro	quasiPart() XXX In order to enable optValue to be recovered from tagCode and source, we need four types rather than the current two: QuasiOpen and QuasiClose.
protected void	skipWhiteSpace() Skips whitespace characters except for newlines.

Methods inherited from class org.quasiliteral.syntax.BaseLexer

charConstant, charLiteral, closeBracket, composite, docComment, endSpan, endToken, isEndOfFile, isJavaIdPart, isJavaIdStart, isWhite, leafEOL, leafTag, needMore, nextChar, nextTopLevelUnit, numberLiteral, openBracket, openBracket, peekChar, peekChar, reset, setSource, skipLine, startToken, stopToken, stringLiteral, syntaxError, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

FOR_ASTS

public static final AstroBuilder FOR_ASTS

Enabled:

FOR_TERMS

public static final AstroBuilder FOR_TERMS

Enabled:

Constructor Detail

ELexer

public ELexer(LineFeeder optLineFeeder,
              boolean partialFlag,
              boolean noTabsFlag)
       throws IOException

Enabled:

ELexer

public ELexer(LineFeeder optLineFeeder,
              boolean partialFlag,
              boolean noTabsFlag,
              AstroBuilder builder)
       throws IOException

Enabled:

Method Detail

make

public static ELexer make(Twine sourceCode,
                          boolean quasiFlag,
                          boolean noTabsFlag)
                   throws IOException

Enabled: @param sourceCode The source code itself

Parameters:

quasiFlag - Should doubled @ and $ be collapsed to singles?

noTabsFlag - Should tabs be rejected as valid whitespace? Assumed to correspond to the e.enable.notabs property.

IOException

isRestBlank

protected boolean isRestBlank(int start)

Consider '#' and '//' comments to be blank as well.

Overrides:

isRestBlank in class BaseLexer

continuer

private Astro continuer(Astro result,
                        int continueCount)

Separated out for use by '>'

nextToken

public Astro nextToken()
                throws IOException,
                       SyntaxException

Enabled: @return :Leaf (see AstroBuilder

Specified by:

nextToken in interface LexerFace

Overrides:

nextToken in class BaseLexer

IOException

SyntaxException

getNextToken

protected Astro getNextToken()
                      throws IOException,
                             SyntaxException

Specified by:

getNextToken in class BaseLexer

IOException

SyntaxException

optKeywordType

private short optKeywordType(String name)

If 'name' is a keyword, return it's token tag code, else -1.

Note that E keywords are case insensitive, so 'name' is first toLowerCase()d.

isIdentifierStart

public static boolean isIdentifierStart(char ch)

Enabled: The first character of an E identifier may be anything accepted as the first character of a Java identifier except '$'. I.e., a letter or '_'.

See Also:

java.lang.Character#isJavaIdentifierStart

isIdentifierPart

public static boolean isIdentifierPart(char ch)

Enabled: A non-first character of an E identifier may be anything accepted as a non-first character of a Java identifier except '$'. The ascii subset consists of letters, digits, and '_'. See Character.isJavaIdentifierPart(char) for the full spec.

isIdentifierOrKeyword

public static boolean isIdentifierOrKeyword(String str)

Enabled: A legal E identifier or keyword is a string whose first character isIdentifierStart, the rest of whose characters are isIdentifierPart.

     <identifier> ::= <idStart> <idPart>*
 

identifier

private Astro identifier()
                  throws IOException,
                         SyntaxException

Called with myChar as the first character of the identifier.

     <identifier> ::= <idStart> <idPart>*
 

IOException

SyntaxException

quasiPart

private Astro quasiPart()
                 throws IOException,
                        SyntaxException

XXX In order to enable optValue to be recovered from tagCode and source, we need four types rather than the current two: QuasiOpen and QuasiClose.

     <nonender> ::= any character but '$', '@', or '`'
     <qconst> ::= <nonender> | "$$" | "@@"
     |            "$\" <charEscape>
     <quasiFull> ::= "`" <qconst>* ("$" | "@")? "`"
     <quasiOpen> ::= "`" <qconst>* ("$" | "@")
     <quasiMid>  ::=     <qconst>* ("$" | "@")
     <quasiClose> ::=    <qconst>* ("$" | "@")? "`"
 

IOException

SyntaxException

optUri

private Astro optUri()
              throws IOException,
                     SyntaxException

Eat a URI or a URIStart.

Assumes the '<' is already eaten and myChar is the first character of the protocol identifier. If the identifier is not immediately followed by a ":", return null and cause no side effects -- in particular, do not effect the current position.

     <uri> ::= '<' <identifier>
  ':' <uric>* '>'
     <uriStart> ::= '<' <identifier> ':'
 

IOException

SyntaxException

skipWhiteSpace

protected void skipWhiteSpace()
                       throws IOException

Description copied from class: BaseLexer

Skips whitespace characters except for newlines.

Overrides:

skipWhiteSpace in class BaseLexer

IOException

main

public static void main(String[] args)
                 throws IOException,
                        SyntaxException

Enabled: Just for testing. Reads an input file and prints one token per line to stdout.

IOException

SyntaxException