/ data / common-syntax

Common Lexing Spec

Common Character Classes

hspace

::=

' ' | '\t'

' ' | '\t'

whitespace

::=

' ' | '\t' | '\n'

' ' | '\t' | '\n'

digit10

::=

isDigit

'0'..'9'

digit8

::=

'0'..'7'

digit16

::=

'0'..'9' | 'a'..'f' | 'A'..'F'

uric

::=

  IETF-URICs
| '\\' | '|' | '#'

  'a'..'z' | 'A'..'Z' | '0'..'9'
| anyof("_$.-;/?:@&=+,!~*'()%\\|#")

The non-ascii cases are not yet tested in the current implementations.

In the uric production, each '\\' (backslash) character is converted to '/', and each '|' (vertical bar) character is converted to ':'. Therefore, the possible semantic values associated with this production do not include the backslash or vertical bar characters.

Common Token Types

In the following table, the bold names with initial capitals are the token types. The others are supporting productions.

digit10s

 ::= 

digit10 ('_'? digit10)*

Integer

 ::= 

  '-'? '0x' digit16 ('_'? digit16)*
| '-'? '0' ('_'? digit8)*  # not yet implemented
/ '-'? digit10s

wholePart

::=

'-'? digit10s

fraction

::=

'.' digit10s

exponent

 ::= 

('e' | 'E') '-'? digit10s

Real64

::=

  wholePart fraction exponent?
| wholePart fraction? exponent

charConst

::=

  '\\' anyof("btnfr\"'\\")
| '\\x' digit16*2 # not yet implemented
| !'\'' !'"' .

Char

 ::= 

'\'' (charConst | '"') '\''

String

::=

'"' (charConst
     | '\'' 
     | '\\' '\n'
    )* '"'

In a literal string, a backslash followed by a newline is ignored -- the backslash eats the newline.

Note that Real64 includes both 0.0 and -0.0. These are distinct, even though they represent the same real number.

Rationale

We allow '_' (underbar) characters within digit sequences so that long digit sequences can be broken up for readability. For example, the number of cents in 1.3 million dollars can be written as "1_300_000_00". (Is it PERL that also allows this?)

For convenience, we allow but do not require single quotes to be escaped in double quoted literals, and vice versa.

For convenience, we allow multi-line string literals without per-line delimeters, even though reviewers can become confused about what they're looking at. Syntax highlighting SHOULD be used to make literals visibly distinct from non-literal source text during reviews.