Skip to content

Commit ce6a617

Browse files
committed
[lex.pptoken] Separate out keyword placeholders
The _preprocessing-token_ term is defined as one of another set of grammar productions, with the three placeholder tokens standing out as specific single tokens. Moving the placeholder tokens into their own production simplifies both the _preprocessing-token_ and _keyword_ grammar productions, plus it guarantees a consistency for the set of placeholder between both places. It also makes it easier for [cpp] to directly cross-reference where needed. Pedantically, the term placeholder token itself is problematic as tokens are a phase 7 concern, where they become keywords. Within the earlier phases where these placeholders are used the correct term is a preprocessing token, so strictly these should be placeholder preprocessing tokens, which is a mouthful. So renaming the term to placeholder keyword, which is hopefully less of a concern as there are no keywords in phase 3--6 to confuse with phase 7 keywords. Finally, clean up the informative list in text of the various kinds of preprocessing tokens, rendering as a list in the source (one kind per line) and adding a cross-reference to each such item.
1 parent 1d9fdb0 commit ce6a617

File tree

2 files changed

+44
-30
lines changed

2 files changed

+44
-30
lines changed

source/lex.tex

Lines changed: 42 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -534,29 +534,30 @@
534534
\indextext{token!preprocessing|(}%
535535
\begin{bnf}
536536
\nontermdef{preprocessing-token}\br
537+
keyword-placeholder\br
537538
header-name\br
538-
import-keyword\br
539-
module-keyword\br
540-
export-keyword\br
541-
identifier\br
542539
pp-number\br
540+
preprocessing-op-or-punc\br
541+
identifier\br
543542
character-literal\br
544543
user-defined-character-literal\br
545544
string-literal\br
546545
user-defined-string-literal\br
547-
preprocessing-op-or-punc\br
548546
\textnormal{each non-whitespace character that cannot be one of the above}
549547
\end{bnf}
550548

551549
\pnum
552550
A preprocessing token is the minimal lexical element of the language in translation
553551
phases 3 through 5.
554-
The categories of preprocessing token are: header names,
555-
placeholder tokens produced by preprocessing \tcode{import} and \tcode{module} directives
556-
(\grammarterm{import-keyword}, \grammarterm{module-keyword}, and \grammarterm{export-keyword}),
557-
identifiers, preprocessing numbers, character literals (including user-defined character
558-
literals), string literals (including user-defined string literals), preprocessing
559-
operators and punctuators, and single non-whitespace characters that do not lexically
552+
The categories of preprocessing token are:
553+
keyword plaeholders\iref{lex.placeholder},
554+
header names\iref{lex.header},
555+
preprocessing numbers\iref{lex.ppnumber},
556+
preprocessing operators and punctuators\iref{lex.operators},
557+
identifiers\iref{lex.name},
558+
character literals\iref{lex.ccon} including user-defined character literals,
559+
string literals\iref{lex.string} including user-defined string literals,
560+
and single non-whitespace characters that do not lexically
560561
match the other preprocessing token categories.
561562
If a \unicode{0027}{apostrophe}, a \unicode{0022}{quotation mark},
562563
or any character not in the basic character set
@@ -587,20 +588,6 @@
587588
shall have the lexical form of a keyword, an identifier, a literal,
588589
or an operator or punctuator.
589590

590-
\pnum
591-
\indexgrammar{\idxgram{import-keyword}}%
592-
\indexgrammar{\idxgram{module-keyword}}%
593-
\indexgrammar{\idxgram{export-keyword}}%
594-
The \grammarterm{import-keyword} is produced
595-
by processing an \keyword{import} directive\iref{cpp.import},
596-
the \grammarterm{module-keyword} is produced
597-
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
598-
the \grammarterm{export-keyword} is produced
599-
by preprocessing either of the previous two directives.
600-
\begin{note}
601-
None has any observable spelling.
602-
\end{note}
603-
604591
\pnum
605592
If the input stream has been parsed into preprocessing tokens up to a
606593
given character:
@@ -687,6 +674,35 @@
687674
\end{example}
688675
\indextext{token!preprocessing|)}
689676

677+
\rSec1[lex.placeholder]{Keyword placeholders}
678+
\indextext{placeholder!keyword|(}%
679+
\indexgrammar{\idxgram{import-keyword}}%
680+
\indexgrammar{\idxgram{module-keyword}}%
681+
\indexgrammar{\idxgram{export-keyword}}%
682+
683+
\begin{bnf}
684+
\nontermdef{keyword-placeholder}\br
685+
export-keyword\br
686+
import-keyword\br
687+
module-keyword
688+
\end{bnf}
689+
690+
\pnum
691+
Keyword placeholders are produced by preprocessing \keyword{import}
692+
and \keyword{module} directives.
693+
The \grammarterm{import-keyword} is produced
694+
by preprocessing an \keyword{import} directive\iref{cpp.import},
695+
the \grammarterm{module-keyword} is produced
696+
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
697+
the \grammarterm{export-keyword} is produced
698+
by preprocessing either of the previous two directives.
699+
700+
\pnum
701+
\begin{note}
702+
Keyword placeholders do not have any observable spelling.
703+
\end{note}
704+
\indextext{placeholder!keyword|(}%
705+
690706
\rSec1[lex.header]{Header names}
691707

692708
\indextext{header!name|(}%
@@ -1012,9 +1028,7 @@
10121028
\begin{bnf}
10131029
\nontermdef{keyword}\br
10141030
\textnormal{any identifier listed in \tref{lex.key}}\br
1015-
\grammarterm{import-keyword}\br
1016-
\grammarterm{module-keyword}\br
1017-
\grammarterm{export-keyword}
1031+
\grammarterm{keyword-placeholder}
10181032
\end{bnf}
10191033

10201034
\pnum

source/preprocessor.tex

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1264,7 +1264,7 @@
12641264
\pnum
12651265
The \tcode{module} and \tcode{export} (if it exists) preprocessing tokens
12661266
are replaced by the \grammarterm{module-keyword} and
1267-
\grammarterm{export-keyword} preprocessing tokens respectively.
1267+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
12681268
\begin{note}
12691269
This makes the line no longer a directive
12701270
so it is not removed at the end of phase 4.
@@ -1317,7 +1317,7 @@
13171317
In all three forms of \grammarterm{pp-import},
13181318
the \tcode{import} and \tcode{export} (if it exists) preprocessing tokens
13191319
are replaced by the \grammarterm{import-keyword} and
1320-
\grammarterm{export-keyword} preprocessing tokens respectively.
1320+
\grammarterm{export-keyword} keyword placeholders respectively\iref{lex.placeholder}.
13211321
\begin{note}
13221322
This makes the line no longer a directive
13231323
so it is not removed at the end of phase 4.

0 commit comments

Comments
 (0)