org.erights.e.elang.syntax
Class ELexer

java.lang.Object
  |
  +--org.quasiliteral.syntax.BaseLexer
        |
        +--org.erights.e.elang.syntax.ELexer
All Implemented Interfaces:
LexerFace, Marker, PassByProxy

public class ELexer
extends BaseLexer

Safe: Breaks textually input into a stream of tokens according to the E language's defined grammar.

Author:
Mark S. Miller

Field Summary
static AstroBuilder FOR_ASTS
          Enabled:
static AstroBuilder FOR_TERMS
          Enabled:
 
Fields inherited from class org.quasiliteral.syntax.BaseLexer
myBuilder, myChar, myContinueCount, myDelayedNextChar, myIndenter, myLData, myLTwine, myNoTabsFlag, myOptStartPos, myOptStartText, myPos, myQuasiFlag
 
Fields inherited from interface org.quasiliteral.syntax.LexerFace
EOFCHAR, EOFTOK
 
Fields inherited from interface org.erights.e.elib.serial.PassByProxy
HONORARY, HONORED_NAMES
 
Constructor Summary
ELexer(LineFeeder optLineFeeder, boolean partialFlag, boolean noTabsFlag)
          Enabled:
ELexer(LineFeeder optLineFeeder, boolean partialFlag, boolean noTabsFlag, AstroBuilder builder)
          Enabled:
 
Method Summary
private  Astro continuer(Astro result, int continueCount)
          Separated out for use by '>'
protected  Astro getNextToken()
           
private  Astro identifier()
          Called with myChar as the first character of the identifier.
static boolean isIdentifierOrKeyword(String str)
          Enabled: A legal E identifier or keyword is a string whose first character isIdentifierStart, the rest of whose characters are isIdentifierPart.
static boolean isIdentifierPart(char ch)
          Enabled: A non-first character of an E identifier may be anything accepted as a non-first character of a Java identifier except '$'.
static boolean isIdentifierStart(char ch)
          Enabled: The first character of an E identifier may be anything accepted as the first character of a Java identifier except '$'.
protected  boolean isRestBlank(int start)
          Consider '#' and '//' comments to be blank as well.
static void main(String[] args)
          Enabled: Just for testing.
static ELexer make(Twine sourceCode, boolean quasiFlag, boolean noTabsFlag)
          Enabled: @param sourceCode The source code itself
 Astro nextToken()
          Enabled: @return :Leaf (see AstroBuilder
private  short optKeywordType(String name)
          If 'name' is a keyword, return it's token tag code, else -1.
private  Astro optUri()
          Eat a URI or a URIStart.
private  Astro quasiPart()
          XXX In order to enable optValue to be recovered from tagCode and source, we need four types rather than the current two: QuasiOpen and QuasiClose.
protected  void skipWhiteSpace()
          Skips whitespace characters except for newlines.
 
Methods inherited from class org.quasiliteral.syntax.BaseLexer
charConstant, charLiteral, closeBracket, composite, docComment, endSpan, endToken, isEndOfFile, isJavaIdPart, isJavaIdStart, isWhite, leafEOL, leafTag, needMore, nextChar, nextTopLevelUnit, numberLiteral, openBracket, openBracket, peekChar, peekChar, reset, setSource, skipLine, startToken, stopToken, stringLiteral, syntaxError, toString
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

FOR_ASTS

public static final AstroBuilder FOR_ASTS
Enabled:


FOR_TERMS

public static final AstroBuilder FOR_TERMS
Enabled:

Constructor Detail

ELexer

public ELexer(LineFeeder optLineFeeder,
              boolean partialFlag,
              boolean noTabsFlag)
       throws IOException
Enabled:


ELexer

public ELexer(LineFeeder optLineFeeder,
              boolean partialFlag,
              boolean noTabsFlag,
              AstroBuilder builder)
       throws IOException
Enabled:

Method Detail

make

public static ELexer make(Twine sourceCode,
                          boolean quasiFlag,
                          boolean noTabsFlag)
                   throws IOException
Enabled: @param sourceCode The source code itself

Parameters:
quasiFlag - Should doubled @ and $ be collapsed to singles?
noTabsFlag - Should tabs be rejected as valid whitespace? Assumed to correspond to the e.enable.notabs property.
IOException

isRestBlank

protected boolean isRestBlank(int start)
Consider '#' and '//' comments to be blank as well.

Overrides:
isRestBlank in class BaseLexer

continuer

private Astro continuer(Astro result,
                        int continueCount)
Separated out for use by '>'


nextToken

public Astro nextToken()
                throws IOException,
                       SyntaxException
Enabled: @return :Leaf (see AstroBuilder

Specified by:
nextToken in interface LexerFace
Overrides:
nextToken in class BaseLexer
IOException
SyntaxException

getNextToken

protected Astro getNextToken()
                      throws IOException,
                             SyntaxException
Specified by:
getNextToken in class BaseLexer
IOException
SyntaxException

optKeywordType

private short optKeywordType(String name)
If 'name' is a keyword, return it's token tag code, else -1.

Note that E keywords are case insensitive, so 'name' is first toLowerCase()d.


isIdentifierStart

public static boolean isIdentifierStart(char ch)
Enabled: The first character of an E identifier may be anything accepted as the first character of a Java identifier except '$'. I.e., a letter or '_'.

See Also:
java.lang.Character#isJavaIdentifierStart

isIdentifierPart

public static boolean isIdentifierPart(char ch)
Enabled: A non-first character of an E identifier may be anything accepted as a non-first character of a Java identifier except '$'. The ascii subset consists of letters, digits, and '_'. See Character.isJavaIdentifierPart(char) for the full spec.


isIdentifierOrKeyword

public static boolean isIdentifierOrKeyword(String str)
Enabled: A legal E identifier or keyword is a string whose first character isIdentifierStart, the rest of whose characters are isIdentifierPart.
     <identifier> ::= <idStart> <idPart>*
 


identifier

private Astro identifier()
                  throws IOException,
                         SyntaxException
Called with myChar as the first character of the identifier.
     <identifier> ::= <idStart> <idPart>*
 

IOException
SyntaxException

quasiPart

private Astro quasiPart()
                 throws IOException,
                        SyntaxException
XXX In order to enable optValue to be recovered from tagCode and source, we need four types rather than the current two: QuasiOpen and QuasiClose.
     <nonender> ::= any character but '$', '@', or '`'
     <qconst> ::= <nonender> | "$$" | "@@"
     |            "$\" <charEscape>
     <quasiFull> ::= "`" <qconst>* ("$" | "@")? "`"
     <quasiOpen> ::= "`" <qconst>* ("$" | "@")
     <quasiMid>  ::=     <qconst>* ("$" | "@")
     <quasiClose> ::=    <qconst>* ("$" | "@")? "`"
 

IOException
SyntaxException

optUri

private Astro optUri()
              throws IOException,
                     SyntaxException
Eat a URI or a URIStart.

Assumes the '<' is already eaten and myChar is the first character of the protocol identifier. If the identifier is not immediately followed by a ":", return null and cause no side effects -- in particular, do not effect the current position.

     <uri> ::= '<' <identifier>
  ':' <uric>* '>'
     <uriStart> ::= '<' <identifier> ':'
 

IOException
SyntaxException

skipWhiteSpace

protected void skipWhiteSpace()
                       throws IOException
Description copied from class: BaseLexer
Skips whitespace characters except for newlines.

Overrides:
skipWhiteSpace in class BaseLexer
IOException

main

public static void main(String[] args)
                 throws IOException,
                        SyntaxException
Enabled: Just for testing. Reads an input file and prints one token per line to stdout.

IOException
SyntaxException


comments?