Accept silent spaces and bar separators, and report unknown commands

[panyam]

What changes

A standalone _ is a silent space. | and || inside a role line are ignored rather than rejected. An unknown command is reported through the errors array instead of thrown out of load().

All three came out of the docs audit. Three of the seven constructs in issue 27 are now answered, and the two open questions that issue raised about underscores turned out to be already settled in src/utils.ts.

Fixes #17, fixes #19, fixes #28.

Prerequisite knowledge

• DSL syntax reference, whose Spaces table has documented _ as a silent space the whole time. This PR makes that true and adds a Bar Separators section next to Whitespace.
• Issue 27, which is the per-construct analysis this PR works through the cheap end of.
• src/utils.ts:107, parseSyllable, which is where the project already wrote down what _ means. It builds new Space(ONE, true) for _ and new Space(ONE, false) for ,, one unit each.

Reviewer's guide

A theatre programme can list a scene the company never staged, and nobody notices until someone turns up expecting to watch it. The grammar here has had Space -> UNDER_SCORE { newSilentSpace } in it from the beginning, and the handler behind it builds a perfectly good silent space, but no %token UNDER_SCORE was ever declared. The production sat there unreachable, _ fell through to the identifier rule, and a silent space rendered as the literal text _. So the interesting part of this change is not writing new behaviour, it is connecting behaviour that was already written and already documented. The same shape shows up twice more. Bar separators are drawn by the renderer from the cycle and were rejected by the tokenizer when typed, and createCommand already knew how to describe an unknown command and threw the description instead of filing it.

Read in this order:

1. src/parser.ts, three edits. The UNDER_SCORE token is the scene that was in the programme. Check the lookahead hardest, since it is what keeps the change non-breaking.
2. src/tests/parser.spec.ts, acceptance for the two grammar edits. Three of its tests pass on master too, deliberately, because they are the guards on what must not change.
3. src/tests/regressions.spec.ts, acceptance for the command errors.
4. docs/tests/playground-patterns.spec.ts, the one existing assertion that changes.
5. docs/content/reference/syntax/index.html, correcting two statements this makes false and documenting bars.

How it works

%token  UNDER_SCORE   /_(?!{IdentChar})/     declared above IDENT
%skip                 /\|\|?/

_ is itself an IdentChar, so the lookahead is what separates a standalone underscore from a run or a word. At the head of ___, the lookahead fails, IDENT matches all three characters and the run stays a literal. At the head of _ R, the lookahead succeeds and the token wins. tlex resolves these by declaration order at equal priority, which is why UNDER_SCORE sits above IDENT rather than below it.

The skip rule cannot reach inside \cycle("|4|2|2|"), because the tokenizer matches the whole quoted run as one STRING at the opening quote and never offers the interior to another rule. There is a test asserting the cycle spec survives, since that is the one way this change could quietly break a document that parses.

For unknown commands:

createCommand(name, params):
    ...known names...
    else:
        errors.push(ParseError("Invalid command: " + name, "InvalidCommand", name))
        return null

newCommandWithBlock:
    if innerCommand is null: return null        # the block goes with it
appendCommand:
    if command is not null: addCommand(command)

endBlock splices a block's commands out of this.commands before newCommandWithBlock runs, so returning null there drops the block rather than leaving its contents stranded at the top level. There is a test for that.

Decision log

• Guarded the underscore token with a lookahead rather than declaring /_/ plain. Plain /_/ also works and reads more simply, and it turns ___ into three silent spaces, which would change the meaning of any document using ___ as a label. One _ per silent space is what parseSyllable already does and what the syntax reference table already said, so _ _ _ is the spelling and the lookahead keeps runs out of it.
• Left IdentChar alone. Excluding _ from it would make the grammar agree with parseSyllable, which splits S_N_D into five atoms, but it would also break every literal anyone has written with an underscore in it. The divergence is worth its own issue rather than a silent change here.
• Skipped bars rather than lexing them. Covered in issue 19 and chosen deliberately: a skipped bar cannot be validated against the cycle, and validation needs a marker atom in the model plus a layout-time pass. That is a feature, and this is a grammar fix.
• Dropped the block attached to an unknown command rather than keeping its commands. The document is already in error, and hoisting a block's contents to the top level would change where those atoms land.
• Did not restore ___ and | to the docs examples. They were rewritten in Make every docs example parse, and add a test that keeps it that way #29 to work around exactly these two gaps, and putting them back is a separate change that this PR does not need in order to be correct.

Risk / blast radius

• Affects every caller of parse() and load(). The grammar edits are additive, and the guards below show the three inputs that could have changed meaning and do not.
• createCommand is public and its return type widens to Command | null. A caller calling it directly and not checking would now get a null where it used to get an exception. It has a doc comment stating the contract.
• Tests covering this: Silent Space Tests and Bar Separator Tests in src/tests/parser.spec.ts, Invalid Commands in src/tests/regressions.spec.ts. Suite goes from 473 to 489 passing. 20 assertions added, 1 replaced, none weakened.
• Be paranoid about the cycle spec. The skip rule and the string token both see |, and the only reason they do not collide is that the tokenizer never looks inside a matched STRING. Bars inside the cycle spec still define the bars is the test that would catch it, and the before/after below shows [4,2,2] intact.
• Be paranoid about the lookahead. _ being an IdentChar is the whole reason it is needed, and if IdentChar ever changes, this token's behaviour changes with it.
• Related tickets: issue 27 loses three of its seven rows and should be updated rather than closed. Issue 18, splitting carnatic out of parser.ts, touches the same file's imports. Issue 30 is unaffected.

Before / after

Captured by building lib/cjs on master, running the script, then again on this branch.

                        BEFORE (master)                      AFTER (this branch)
Sw: S _ R               lit(S)  lit(_)  lit(R)               lit(S)  silent  lit(R)
Sw: S ___ R             lit(S)  lit(___)  lit(R)             lit(S)  lit(___)  lit(R)
Sw: S_N_D               lit(S_N_D)                           lit(S_N_D)
Sw: S R | G M ||        ParseError: Unexpected Character: |  lit(S) lit(R) lit(G) lit(M)
\nosuchcommand          THREW Invalid command: nosuchcommand ParseError: Invalid command: nosuchcommand

Rows two and three are the point of the lookahead. They are identical in both columns, and there are tests pinning them that pass on master as well as here.

The cycle spec, which is the thing the skip rule had to leave alone:

\cycle("|4|2|2|")  with bars in the role line
  cycle bars: [4,2,2]      cycle duration: 8/1

Out of scope

• Validating a bar against the cycle, which is what would make typing one worth doing. Issue 19 covers the reasoning.
• The divergence between parseSyllable("S_N_D"), which splits into five atoms, and the grammar, which keeps one literal. Worth a follow-up.
• The remaining four constructs in issue 27: sharps, free-time sections, paren gamaka grouping and floating embellishment operators. Those are design questions before they are grammar questions.
• Restoring ___ and | to the docs examples.