lex.dd

Ddoc

$(SPEC_S Lexical,

	In D, the lexical analysis is independent of the syntax parsing and the
	semantic analysis. The lexical analyzer splits the source text up into
	tokens. The lexical grammar describes what those tokens are. The D
	lexical grammar is designed to be suitable for high speed scanning, it
	has a minimum of special case rules, there is only one phase of
	translation, and to make it easy to write a correct scanner
	for. The tokens are readily recognizable by those familiar with C and
	C++.

<h3>Phases of Compilation</h3>

	The process of compiling is divided into multiple phases. Each phase
	has no dependence on subsequent phases. For example, the scanner is
	not perturbed by the semantic analyzer. This separation of the passes
	makes language tools like syntax 
	directed editors relatively easy to produce.
	It also is possible to compress D source by storing it in
	$(SINGLEQUOTE tokenized) form.

$(OL
	$(LI $(B source character set)$(BR)

	The source file is checked to see what character set it is,
	and the appropriate scanner is loaded. ASCII and UTF
	formats are accepted.
	)

	$(LI $(B script line) $(BR)

	If the first line starts with $(GREEN #!) then the first line
	is ignored.
	)

	$(LI $(B lexical analysis)$(BR)

	The source file is divided up into a sequence of tokens.
	$(LINK2 #specialtokens, Special tokens) are replaced with other tokens.
	$(LINK2 #specialtokenseq, Special token sequences)
	are processed and removed.
	)

	$(LI $(B syntax analysis)$(BR)

	The sequence of tokens is parsed to form syntax trees.
	)

	$(LI $(B semantic analysis)$(BR)

	The syntax trees are traversed to declare variables, load symbol tables, assign 
	types, and in general determine the meaning of the program.
	)

	$(LI $(B optimization)$(BR)

	Optimization is an optional pass that tries to rewrite the program
	in a semantically equivalent, but faster executing, version.
	)

	$(LI $(B code generation)$(BR)

	Instructions are selected from the target architecture to implement
	the semantics of the program. The typical result will be
	an object file, suitable for input to a linker.
	)
)


<h3>Source Text</h3>

	D source text can be in one of the following formats:

	$(UL 
	$(LI ASCII)
	$(LI UTF-8)
	$(LI UTF-16BE)
	$(LI UTF-16LE)
	$(LI UTF-32BE)
	$(LI UTF-32LE)
	)

	UTF-8 is a superset of traditional 7-bit ASCII.
	One of the
	following UTF BOMs (Byte Order Marks) can be present at the beginning
	of the source text:
	<p>

	$(TABLE2 UTF Byte Order Marks,
	$(TR
	$(TH Format)
	$(TH BOM)
	)
	$(TR
	$(TD UTF-8)
	$(TD EF BB BF)
	)
	$(TR
	$(TD UTF-16BE)
	$(TD FE FF)
	)
	$(TR
	$(TD UTF-16LE)
	$(TD FF FE)
	)
	$(TR
	$(TD UTF-32BE)
	$(TD 00 00 FE FF)
	)
	$(TR
	$(TD UTF-32LE)
	$(TD FF FE 00 00)
	)
	$(TR
	$(TD ASCII)
	$(TD no BOM)
	)
	)

	$(P If the source file does not start with a BOM, then the first
	character must be less than or equal to U0000007F.)

	$(P There are no digraphs or trigraphs in D.)

	$(P The source text is decoded from its source representation
	into Unicode $(I Character)s.
	The $(I Character)s are further divided into:

	$(LINK2 #whitespace, white space),
	$(LINK2 #endofline, end of lines),
	$(LINK2 #comment, comments),
	$(LINK2 #specialtokens, special token sequences),
	$(LINK2 #tokens, tokens),
	all followed by $(LINK2 #eof, end of file).
	)

	$(P The source text is split into tokens using the maximal munch
	technique, i.e., the
	lexical analyzer tries to make the longest token it can. For example
	<code>&gt;&gt;</code> is a right shift token,
	not two greater than tokens. An exception to this rule is that a ..
	embedded inside what looks like two floating point literals, as in
	1..2, is interpreted as if the .. was separated by a space from the
	first integer.
	)

<h3>$(LNAME2 eof, End of File)</h3>

$(GRAMMAR
$(I EndOfFile):
	$(I physical end of the file)
	\u0000
	\u001A
)

	The source text is terminated by whichever comes first.

<h3>$(LNAME2 endofline, End of Line)</h3>

$(GRAMMAR
$(I EndOfLine):
	\u000D
	\u000A
	\u000D \u000A
	$(I EndOfFile)
)

	There is no backslash line splicing, nor are there any limits
	on the length of a line.

<h3>$(LNAME2 whitespace, White Space)</h3>

$(GRAMMAR
$(I WhiteSpace):
	$(I Space)
	$(I Space) $(I WhiteSpace)

$(I Space):
	\u0020
	\u0009
	\u000B
	\u000C
)


<h3>$(LNAME2 comment, Comments)</h3>

$(GRAMMAR
$(I Comment):
	$(B /*) $(I Characters) $(B */)
	$(B //) $(I Characters) $(I EndOfLine)
	$(I NestingBlockComment)

$(I Characters):
	$(I Character)
	$(I Character) $(I Characters)

$(I NestingBlockComment):
	$(B /+) $(I NestingBlockCommentCharacters) $(B +/)

$(I NestingBlockCommentCharacters):
	$(I NestingBlockCommentCharacter)
	$(I NestingBlockCommentCharacter) $(I NestingBlockCommentCharacters)

$(I NestingBlockCommentCharacter):
	$(I Character)
	$(I NestingBlockComment)
)

	D has three kinds of comments:
	$(OL 
	$(LI Block comments can span multiple lines, but do not nest.)
	$(LI Line comments terminate at the end of the line.)
	$(LI Nesting comments can span multiple lines and can nest.)
	)

	$(P
	The contents of strings and comments are not tokenized.  Consequently,
	comment openings occurring within a string do not begin a comment, and
	string delimiters within a comment do not affect the recognition of
	comment closings and nested "/+" comment openings.  With the exception
	of "/+" occurring within a "/+" comment, comment openings within a
	comment are ignored.
	)

-------------
a = /+ // +/ 1;		// parses as if 'a = 1;'
a = /+ "+/" +/ 1";	// parses as if 'a = " +/ 1";'
a = /+ /* +/ */ 3;	// parses as if 'a = */ 3;'
-------------

	Comments cannot be used as token concatenators, for example,
	<code>abc/**/def</code> is two tokens, $(TT abc) and $(TT def),
	not one $(TT abcdef) token.

<h3>$(LNAME2 tokens, Tokens)</h3>

$(GRAMMAR
$(I Token):
	$(LINK2 #identifier, $(I Identifier))
	$(LINK2 #StringLiteral, $(I StringLiteral))
	$(LINK2 #characterliteral, $(I CharacterLiteral))
	$(LINK2 #integerliteral, $(I IntegerLiteral))
	$(LINK2 #floatliteral, $(I FloatLiteral))
	$(LINK2 #keyword, $(I Keyword))
	$(B /)
	$(B /=)
	$(B .)
	$(B ..)
	$(B ...)
	$(B &)
	$(B &=)
	$(B &&)
	$(B |)
	$(B |=)
	$(B ||)
	$(B -)
	$(B -=)
	$(B --)
	$(B +)
	$(B +=)
	$(B ++)
	$(B &lt;)
	$(B &lt;=)
	$(B &lt;&lt;)
	$(B &lt;&lt;=)
	$(B &lt;&gt;)
	$(B &lt;&gt=)
	$(B &gt;)
	$(B &gt;=)
	$(B &gt;&gt;=)
	$(B &gt;&gt;&gt;=)
	$(B &gt;&gt;)
	$(B &gt;&gt;&gt;)
	$(B !)
	$(B !=)
	$(B !&lt;&gt;)
	$(B !&lt;&gt;=)
	$(B !&lt;)
	$(B !&lt;=)
	$(B !&gt;)
	$(B !&gt;=)
	$(B $(LPAREN))
	$(B $(RPAREN))
	$(B [)
	$(B ])
	$(B {)
	$(B })
	$(B ?)
	$(B ,)
	$(B ;)
	$(B :)
	$(B $)
	$(B =)
	$(B ==)
	$(B *)
	$(B *=)
	$(B %)
	$(B %=)
	$(B ^)
	$(B ^=)
	$(B ~)
	$(B ~=)
	$(V2 $(B @))
)

<h3>$(LNAME2 identifier, Identifiers)</h3>

$(GRAMMAR
$(I Identifier):
	$(I IdentiferStart)
	$(I IdentiferStart) $(I IdentifierChars)

$(I IdentifierChars):
	$(I IdentiferChar)
	$(I IdentiferChar) $(I IdentifierChars)

$(I IdentifierStart):
	$(B _)
	$(I Letter)
	$(I UniversalAlpha)

$(I IdentifierChar):
	$(I IdentiferStart)
	$(B 0)
	$(I NonZeroDigit)
)


	Identifiers start with a letter, $(B _), or universal alpha,
	and are followed by any number
	of letters, $(B _), digits, or universal alphas.
	Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D.
	(This is the C99 Standard.)
	Identifiers can be arbitrarily long, and are case sensitive.
	Identifiers starting with $(B __) (two underscores) are reserved.

<h3>$(LNAME2 StringLiteral, String Literals)</h3>

$(GRAMMAR
$(I StringLiteral):
	$(I WysiwygString)
	$(I AlternateWysiwygString)
	$(I DoubleQuotedString)
$(V1
	$(I EscapeSequence))
	$(I HexString)
$(V2
	$(I DelimitedString)
	$(I TokenString))

$(I WysiwygString):
	$(B r") $(I WysiwygCharacters) $(B ") $(I Postfix<sub>opt</sub>)

$(I AlternateWysiwygString):
	$(B `) $(I WysiwygCharacters) $(B `) $(I Postfix<sub>opt</sub>)

$(I WysiwygCharacters):
	$(I WysiwygCharacter)
	$(I WysiwygCharacter) $(I WysiwygCharacters)

$(I WysiwygCharacter):
	$(I Character)
	$(I EndOfLine)

$(I DoubleQuotedString):
	$(B ") $(I DoubleQuotedCharacters) $(B ") $(I Postfix<sub>opt</sub>)

$(I DoubleQuotedCharacters):
	$(I DoubleQuotedCharacter)
	$(I DoubleQuotedCharacter) $(I DoubleQuotedCharacters)

$(I DoubleQuotedCharacter):
	$(I Character)
	$(I EscapeSequence)
	$(I EndOfLine)

$(LNAME2 EscapeSequence, $(I EscapeSequence)):
	$(B \')
	$(B \")
	$(B \?)
	$(B \\)
	$(B \a)
	$(B \b)
	$(B \f)
	$(B \n)
	$(B \r)
	$(B \t)
	$(B \v)
	$(B \) $(I EndOfFile)
	$(B \x) $(I HexDigit) $(I HexDigit)
	$(B \) $(I OctalDigit)
	$(B \) $(I OctalDigit) $(I OctalDigit)
	$(B \) $(I OctalDigit) $(I OctalDigit) $(I OctalDigit)
	$(B \u) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit)
	$(B \U) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit) $(I HexDigit)
	$(B \&amp;) $(LINK2 entity.html, $(I NamedCharacterEntity)) $(B ;)

$(I HexString):
	$(B x") $(I HexStringChars) $(B ") $(I Postfix<sub>opt</sub>)

$(I HexStringChars):
	$(I HexStringChar)
	$(I HexStringChar) $(I HexStringChars)

$(I HexStringChar):
	$(I HexDigit)
	$(I WhiteSpace)
	$(I EndOfLine)

$(I Postfix):
	$(B c)
	$(B w)
	$(B d)

$(V2
$(I DelimitedString):
	$(B q") $(I Delimiter) $(I WysiwygCharacters) $(I MatchingDelimiter) $(B ")

$(I TokenString):
	$(B q{) $(I Tokens) $(B })
)
)

	$(P
	A string literal is either a double quoted string, a wysiwyg quoted
	string, an escape sequence,
	$(V2 a delimited string, a token string,)
	or a hex string.
	)

<h4>Wysiwyg Strings</h4>

	$(P
	Wysiwyg quoted strings are enclosed by r" and ".
	All characters between
	the r" and " are part of the string except for $(I EndOfLine) which is
	regarded as a single \n character.
	There are no escape sequences inside r" ":
	)

---------------
r"hello"
r"c:\root\foo.exe"
r"ab\n"			// string is 4 characters, 'a', 'b', '\', 'n'
---------------

	$(P
	An alternate form of wysiwyg strings are enclosed by backquotes,
	the ` character. The ` character is not available on some keyboards
	and the font rendering of it is sometimes indistinguishable from
	the regular ' character. Since, however, the ` is rarely used,
	it is useful to delineate strings with " in them.
	)

---------------
`hello`
`c:\root\foo.exe`
`ab\n`			// string is 4 characters, 'a', 'b', '\', 'n'
---------------

<h4>Double Quoted Strings</h4>

	Double quoted strings are enclosed by "". Escape sequences can be
	embedded into them with the typical \ notation.
	$(I EndOfLine) is regarded as a single \n character.

---------------
"hello"
"c:\\root\\foo.exe"
"ab\n"			// string is 3 characters, 'a', 'b', and a linefeed
"ab
"			// string is 3 characters, 'a', 'b', and a linefeed
---------------

$(V1
<h4>Escape Strings</h4>

	$(P Escape strings start with a \ and form an escape character sequence.
	Adjacent escape strings are concatenated:
	)

<pre>
\n			the linefeed character
\t			the tab character
\"			the double quote character
\012			octal
\x1A			hex
\u1234			wchar character
\U00101234		dchar character
\&amp;reg;			&reg; dchar character
\r\n			carriage return, line feed
</pre>

	$(P Undefined escape sequences are errors.
	Although string literals are defined to be composed of
	UTF characters, the octal and hex escape sequences allow
	the insertion of arbitrary binary data.
	\u and \U escape sequences can only be used to insert
	valid UTF characters.
	)
)

<h4>Hex Strings</h4>

	$(P Hex strings allow string literals to be created using hex data.
	The hex data need not form valid UTF characters.
	)

--------------
x"0A"			// same as "\x0A"
x"00 FBCD 32FD 0A"	// same as "\x00\xFB\xCD\x32\xFD\x0A"
--------------

	Whitespace and newlines are ignored, so the hex data can be
	easily formatted.
	The number of hex characters must be a multiple of 2.
	<p>

	Adjacent strings are concatenated with the ~ operator, or by simple
	juxtaposition:

--------------
"hello " ~ "world" ~ \n	// forms the string 'h','e','l','l','o',' ',
			// 'w','o','r','l','d',linefeed
--------------

	The following are all equivalent:

-----------------
"ab" "c"
r"ab" r"c"
r"a" "bc"
"a" ~ "b" ~ "c"
\x61"bc"
-----------------

	The optional $(I Postfix) character gives a specific type
	to the string, rather than it being inferred from the context.
	This is useful when the type cannot be unambiguously inferred,
	such as when overloading based on string type. The types corresponding
	to the postfix characters are:
	<p>

	$(TABLE2 String Literal Postfix Characters,
	$(TR
	$(TH Postfix)
	$(TH Type)
	)
	$(TR
	$(TD $(B c))
	$(TD char[ ])
	)
	$(TR
	$(TD $(B w))
	$(TD wchar[ ])
	)
	$(TR
	$(TD $(B d))
	$(TD dchar[ ])
	)
	)

---
"hello"c          // char[]
"hello"w          // wchar[]
"hello"d          // dchar[]
---

	$(P String literals are read only. Writes to string literals
	cannot always be detected, but cause undefined behavior.)

$(V2
<h4>Delimited Strings</h4>

	$(P Delimited strings use various forms of delimiters.
	The delimiter, whether a character or identifer,
	must immediately follow the " without any intervening whitespace.
	The terminating delimiter must immediately precede the closing "
	without any intervening whitespace.
	A $(I nesting delimiter) nests, and is one of the
	following characters:
	)

	$(TABLE2 Nesting Delimiters,
	$(TR
	$(TH Delimiter)
	$(TH Matching Delimiter)
	)
	$(TR
	$(TD [)
	$(TD ])
	)
	$(TR
	$(TD $(LPAREN))
	$(TD $(RPAREN))
	)
	$(TR
	$(TD &lt;)
	$(TD &gt;)
	)
	$(TR
	$(TD {)
	$(TD })
	)
	)

---
q"(foo(xxx))"   // "foo(xxx)"
q"[foo{]"       // "foo{"
---

	$(P If the delimiter is an identifier, the identifier must
	be immediately followed by a newline, and the matching
	delimiter is the same identifier starting at the beginning
	of the line:
	)
---
writefln(q"EOS
This
is a multi-line
heredoc string
EOS"
);
---
	$(P The newline following the opening identifier is not part
	of the string, but the last newline before the closing
	identifier is part of the string.
	)

	$(P Otherwise, the matching delimiter is the same as
	the delimiter character:)

---
q"/foo]/"       // "foo]"
q"/abc/def/"    // error
---

<h4>Token Strings</h4>

	$(P Token strings open with the characters $(B q{) and close with
	the token $(B }). In between must be valid D tokens.
	The $(B {) and $(B }) tokens nest.
	The string is formed of all the characters between the opening
	and closing of the token string, including comments.
	)

---
q{foo}               // "foo"
q{/*}*/ }            // "/*}*/ "
q{ foo(q{hello}); }  // " foo(q{hello}); "
q{ @ }               // error, @ is not a valid D token
q{ __TIME__ }        // " __TIME__ ", i.e. it is not replaced with the time
q{ __EOF__ }         // error, as __EOF__ is not a token, it's end of file
---

)

<h3>$(LNAME2 characterliteral, Character Literals)</h3>

$(GRAMMAR
$(I CharacterLiteral):
	$(B ') $(I SingleQuotedCharacter) $(B ')

$(I SingleQuotedCharacter):
	$(I Character)
	$(I EscapeSequence)
)

	Character literals are a single character or escape sequence
	enclosed by single quotes, ' '.

<h3>$(LNAME2 integerliteral, Integer Literals)</h3>

$(GRAMMAR
$(GNAME IntegerLiteral):
	$(I Integer)
	$(I Integer) $(I IntegerSuffix)

$(I Integer):
	$(I Decimal)
	$(I Binary)
	$(I Octal)
	$(I Hexadecimal)

$(I IntegerSuffix):
	$(B L)
	$(B u)
	$(B U)
	$(B Lu)
	$(B LU)
	$(B uL)
	$(B UL)

$(GNAME Decimal):
	$(B 0)
	$(I NonZeroDigit)
	$(I NonZeroDigit) $(I DecimalDigits)

$(I Binary):
	$(B 0b) $(I BinaryDigits)
	$(B 0B) $(I BinaryDigits)

$(I Octal):
	$(B 0) $(I OctalDigits)

$(I Hexadecimal):
	$(B 0x) $(I HexDigits)
	$(B 0X) $(I HexDigits)

$(I NonZeroDigit):
	$(B 1)
	$(B 2)
	$(B 3)
	$(B 4)
	$(B 5)
	$(B 6)
	$(B 7)
	$(B 8)
	$(B 9)

$(GNAME DecimalDigits):
	$(I DecimalDigit)
	$(I DecimalDigit) $(I DecimalDigits)

$(GNAME DecimalDigit):
	$(B 0)
	$(I NonZeroDigit)
	$(B _)

$(I BinaryDigits):
	$(I BinaryDigit)
	$(I BinaryDigit) $(I BinaryDigits)

$(I BinaryDigit):
	$(B 0)
	$(B 1)
	$(B _)

$(I OctalDigits):
	$(I OctalDigit)
	$(I OctalDigit) $(I OctalDigits)

$(I OctalDigit):
	$(B 0)
	$(B 1)
	$(B 2)
	$(B 3)
	$(B 4)
	$(B 5)
	$(B 6)
	$(B 7)
	$(B _)

$(I HexDigits):
	$(I HexDigit)
	$(I HexDigit) $(I HexDigits)

$(I HexDigit):
	$(I DecimalDigit)
	$(B a)
	$(B b)
	$(B c)
	$(B d)
	$(B e)
	$(B f)
	$(B A)
	$(B B)
	$(B C)
	$(B D)
	$(B E)
	$(B F)
	$(B _)
)

	Integers can be specified in decimal, binary, octal, or hexadecimal.
<p>
	Decimal integers are a sequence of decimal digits.
<p>
	$(LNAME2 binary-literals, Binary integers) are a sequence of binary digits preceded
	by a $(SINGLEQUOTE 0b).
<p>
	Octal integers are a sequence of octal digits preceded by a $(SINGLEQUOTE 0).
<p>
	Hexadecimal integers are a sequence of hexadecimal digits preceded
	by a $(SINGLEQUOTE 0x).
<p>
	Integers can have embedded $(SINGLEQUOTE _) characters, which are ignored.
	The embedded $(SINGLEQUOTE _) are useful for formatting long literals, such
	as using them as a thousands separator:

-------------
123_456		// 123456
1_2_3_4_5_6_	// 123456
-------------

	Integers can be immediately followed by one $(SINGLEQUOTE L) or one
	$(SINGLEQUOTE u) or both.
<p>
	The type of the integer is resolved as follows:
	<p>

	$(TABLE2 Decimal Literal Types,
	$(TR
	$(TH Decimal Literal)
	$(TH Type)
	)
	$(TR
	$(TD 0 .. 2_147_483_647)
	$(TD int)
	)
	$(TR
	$(TD 2_147_483_648 .. 9_223_372_036_854_775_807L)
	$(TD long)
	)
	$(TR
	$(TH Decimal Literal, L Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0L .. 9_223_372_036_854_775_807L)
	$(TD long)
	)
	$(TR
	$(TH Decimal Literal, U Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0U .. 4_294_967_296U)
	$(TD uint)
	)
	$(TR
	$(TD 4_294_967_296U .. 18_446_744_073_709_551_615UL)
	$(TD ulong)
	)
	$(TR
	$(TH Decimal Literal, UL Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0UL .. 18_446_744_073_709_551_615UL)
	$(TD ulong)
	)

	$(TR
	$(TH Non-Decimal Literal)
	$(TH Type)
	)
	$(TR
	$(TD 0x0 .. 0x7FFF_FFFF)
	$(TD int)
	)
	$(TR
	$(TD 0x8000_0000 .. 0xFFFF_FFFF)
	$(TD uint)
	)
	$(TR
	$(TD 0x1_0000_0000 .. 0x7FFF_FFFF_FFFF_FFFF)
	$(TD long)
	)
	$(TR
	$(TD 0x8000_0000_0000_0000 .. 0xFFFF_FFFF_FFFF_FFFF)
	$(TD ulong)
	)
	$(TR
	$(TH Non-Decimal Literal, L Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0x0L .. 0x7FFF_FFFF_FFFF_FFFFL)
	$(TD long)
	)
	$(TR
	$(TD 0x8000_0000_0000_0000L .. 0xFFFF_FFFF_FFFF_FFFFL)
	$(TD ulong)
	)
	$(TR
	$(TH Non-Decimal Literal, U Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0x0U .. 0xFFFF_FFFFU)
	$(TD uint)
	)
	$(TR
	$(TD 0x1_0000_0000UL .. 0xFFFF_FFFF_FFFF_FFFFUL)
	$(TD ulong)
	)
	$(TR
	$(TH Non-Decimal Literal, UL Suffix)
	$(TH Type)
	)
	$(TR
	$(TD 0x0UL .. 0xFFFF_FFFF_FFFF_FFFFUL)
	$(TD ulong)
	)

	)


<h3>$(LNAME2 floatliteral, Floating Literals)</h3>

$(GRAMMAR
$(GNAME FloatLiteral):
	$(I Float)
	$(I Float) $(I Suffix)
	$(I Integer) $(I ImaginarySuffix)
	$(I Integer) $(I FloatSuffix) $(I ImaginarySuffix)
	$(I Integer) $(I RealSuffix) $(I ImaginarySuffix)

$(I Float):
	$(I DecimalFloat)
	$(I HexFloat)

$(I DecimalFloat):
	$(GLINK LeadingDecimal) $(B .)
	$(GLINK LeadingDecimal) $(B .) $(I DecimalDigits)
	$(I DecimalDigits) $(B .) $(I DecimalDigits) $(I DecimalExponent)
	$(B .) $(I Decimal)
	$(B .) $(I Decimal) $(I DecimalExponent)
	$(GLINK LeadingDecimal) $(I DecimalExponent)

$(I DecimalExponent)
	$(B e) $(I DecimalDigits)
	$(B E) $(I DecimalDigits)
	$(B e+) $(I DecimalDigits)
	$(B E+) $(I DecimalDigits)
	$(B e-) $(I DecimalDigits)
	$(B E-) $(I DecimalDigits)

$(I HexFloat):
	$(I HexPrefix) $(I HexDigits) $(B .) $(I HexDigits) $(I HexExponent)
	$(I HexPrefix) $(B .) $(I HexDigits) $(I HexExponent)
	$(I HexPrefix) $(I HexDigits) $(I HexExponent)

$(I HexPrefix):
	$(B 0x)
	$(B 0X)

$(I HexExponent):
	$(B p) $(I DecimalDigits)
	$(B P) $(I DecimalDigits)
	$(B p+) $(I DecimalDigits)
	$(B P+) $(I DecimalDigits)
	$(B p-) $(I DecimalDigits)
	$(B P-) $(I DecimalDigits)

$(I Suffix):
	$(I FloatSuffix)
	$(I RealSuffix)
	$(I ImaginarySuffix)
	$(I FloatSuffix) $(I ImaginarySuffix)
	$(I RealSuffix) $(I ImaginarySuffix)

$(I FloatSuffix):
	$(B f)
	$(B F)

$(I RealSuffix):
	$(B L)

$(I ImaginarySuffix):
	$(B i)

$(GNAME LeadingDecimal):
	$(GLINK Decimal)
	$(B 0) $(GLINK DecimalDigits)
)

	Floats can be in decimal or hexadecimal format,
	as in standard C.
	<p>

	Hexadecimal floats are preceded with a $(B 0x) and the
	exponent is a $(B p)
	or $(B P) followed by a decimal number serving as the exponent
	of 2.
	<p>

	Floating literals can have embedded $(SINGLEQUOTE _) characters, which are ignored.
	The embedded $(SINGLEQUOTE _) are useful for formatting long literals to
	make them more readable, such
	as using them as a thousands separator:

---------
123_456.567_8		// 123456.5678
1_2_3_4_5_6_._5_6_7_8	// 123456.5678
1_2_3_4_5_6_._5e-6_	// 123456.5e-6
---------

	Floating literals with no suffix are of type double.
	Floats can be followed by one $(B f), $(B F),
	or $(B L) suffix.
	The $(B f) or $(B F) suffix means it is a
	float, and $(B L) means it is a real.
	<p>

	If a floating literal is followed by $(B i), then it is an
	$(I ireal) (imaginary) type.
	<p>

	Examples:

---------
0x1.FFFFFFFFFFFFFp1023		// double.max
0x1p-52				// double.epsilon
1.175494351e-38F		// float.min
6.3i				// idouble 6.3
6.3fi				// ifloat 6.3
6.3Li				// ireal 6.3
---------

	It is an error if the literal exceeds the range of the type.
	It is not an error if the literal is rounded to fit into
	the significant digits of the type.
	<p>

	Complex literals are not tokens, but are assembled from
	real and imaginary expressions in the semantic analysis:

---------
4.5 + 6.2i		// complex number
---------

<h3>$(LNAME2 keyword, Keywords)</h3>

	Keywords are reserved identifiers.

$(GRAMMAR
$(I Keyword):
	$(B abstract)
	$(B alias)
	$(B align)
	$(B asm)
	$(B assert)
	$(B auto)

	$(B body)
	$(B bool)
	$(B break)
	$(B byte)

	$(B case)
	$(B cast)
	$(B catch)
	$(B cdouble)
	$(B cent)
	$(B cfloat)
	$(B char)
	$(B class)
	$(B const)
	$(B continue)
	$(B creal)

	$(B dchar)
	$(B debug)
	$(B default)
	$(B delegate)
	$(B delete)
	$(B deprecated)
	$(B do)
	$(B double)

	$(B else)
	$(B enum)
	$(B export)
	$(B extern)

	$(B false)
	$(B final)
	$(B finally)
	$(B float)
	$(B for)
	$(B foreach)
	$(B foreach_reverse)
	$(B function)

	$(B goto)

	$(B idouble)
	$(B if)
	$(B ifloat)
$(V2
	$(B immutable)
)	$(B import)
	$(B in)
	$(B inout)
	$(B int)
	$(B interface)
	$(B invariant)
	$(B ireal)
	$(B is)

	$(B lazy)
	$(B long)

	$(B macro)
	$(B mixin)
	$(B module)

	$(B new)
$(V2
	$(B nothrow)
)	$(B null)

	$(B out)
	$(B override)

	$(B package)
	$(B pragma)
	$(B private)
	$(B protected)
	$(B public)
$(V2
	$(B pure)
)
	$(B real)
	$(B ref)
	$(B return)

	$(B scope)
$(V2
	$(B shared)
)	$(B short)
	$(B static)
	$(B struct)
	$(B super)
	$(B switch)
	$(B synchronized)

	$(B template)
	$(B this)
	$(B throw)
	$(B true)
	$(B try)
	$(B typedef)
	$(B typeid)
	$(B typeof)

	$(B ubyte)
	$(B ucent)
	$(B uint)
	$(B ulong)
	$(B union)
	$(B unittest)
	$(B ushort)

	$(B version)
	$(B void)
	$(B volatile)

	$(B wchar)
	$(B while)
	$(B with)
$(V2 
	$(B __FILE__)
	$(B __LINE__)
	$(B __gshared)
	$(B __thread)
	$(B __traits))
)

<h3>$(LNAME2 specialtokens, Special Tokens)</h3>

	$(P
	These tokens are replaced with other tokens according to the following
	table:
	)

	$(TABLE2 Special Tokens,
	$(TR
	$(TH Special Token)
	$(TH Replaced with...)
	)
$(V1
	$(TR
	$(TD $(B __FILE__))
	$(TD string literal containing source file name)
	)
	$(TR
	$(TD $(B __LINE__))
	$(TD integer literal of the current source line number)
	)
)
	$(TR
	$(TD $(B __DATE__))
	$(TD string literal of the date of compilation "$(I mmm dd yyyy)")
	)
$(V2
	$(TR
	$(TD $(B __EOF__))
	$(TD sets the scanner to the end of the file)
	)
)
	$(TR
	$(TD $(B __TIME__))
	$(TD string literal of the time of compilation "$(I hh:mm:ss)")
	)
	$(TR
	$(TD $(B __TIMESTAMP__))
	$(TD string literal of the date and time of compilation "$(I www mmm dd hh:mm:ss yyyy)")
	)
	$(TR
	$(TD $(B __VENDOR__))
	$(TD Compiler vendor string, such as "Digital Mars D")
	)
	$(TR
	$(TD $(B __VERSION__))
	$(TD Compiler version as an integer, such as 2001)
	)
	)

<h3>$(LNAME2 specialtokenseq, Special Token Sequences)</h3>

	Special token sequences are processed by the lexical analyzer, may
	appear between any other tokens, and do not affect the syntax
	parsing.
	<p>

	There is currently only one special token sequence, $(TT #line).

$(GRAMMAR
$(I SpecialTokenSequence):
	$(B # line) $(I Integer) $(I EndOfLine)
	$(B # line) $(I Integer) $(I Filespec) $(I EndOfLine)

$(I Filespec):
	$(B ") $(I Characters) $(B ")
)

	This sets the source line number to $(I Integer),
	and optionally the source file 	name to $(I Filespec),
	beginning with the next line of source text.
	The source file and line number is used for printing error messages
	and for mapping generated code back to the source for the symbolic
	debugging output.
	<p>

	For example:

-----------------
int #line 6 "foo\bar"
x;			// this is now line 6 of file foo\bar
-----------------

	Note that the backslash character is not treated specially inside
	$(I Filespec) strings.

)

Macros:
	TITLE=Lexical
	WIKI=Lex