Skip to content

Make every docs example parse, and add a test that keeps it that way - #29

Merged
panyam merged 2 commits into
masterfrom
fix/docs-examples-parse
Sep 21, 2026
Merged

panyam merged 2 commits into
masterfrom
fix/docs-examples-parse

Conversation

@panyam

@panyam panyam commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

What changes

Every live <notation> example on the docs site parses now. 39 of 148 did not, and NotationBlock.updatePreview() renders nothing when load() reports errors, so those were blank boxes with the reason only in the browser console. Five more parsed but contained ___, which lexes as an ordinary identifier and renders as visible text where a silent space was meant.

docs/tests/examples.spec.ts asserts both properties for every example and runs in CI with the rest of the suite.

Prerequisite knowledge

  • DSL syntax reference, which is the page that says what the notation is supposed to be. It is also one of the pages being corrected.
  • Embellishments reference, for the gamaka operators. Four examples there had the operator in a position the grammar does not allow.
  • Issue 27, which is where the grammar questions went instead of into this PR.

Reviewer's guide

A restaurant that prints a menu before the kitchen can cook the dishes ends up with two problems, and only one of them is about food. The docs here describe a notation language in some detail, and the parser implements a smaller one. Bar separators, silent spaces, sharps, free-time sections and a paren-based gamaka syntax all appear in the examples and none of them exist in the grammar. So a quarter of the menu arrived as an empty plate, silently, because the renderer draws nothing when a parse fails. The fix is not to teach the kitchen five new dishes in an afternoon, since nobody has decided what those dishes should be. It is to reprint the menu with what the kitchen actually cooks, write down the five missing recipes somewhere they can be argued about properly, and put a checker on the pass so a dish that cannot be cooked never reaches a table again.

Read in this order:

  1. docs/tests/examples.spec.ts, the checker on the pass. Two assertions per example. Worth checking what it deliberately skips, since it drops <!-- --> regions and template bodies, which is what the browser sees. stripComments walks the string rather than running one regex replace, for reasons in the decision log.
  2. docs/content/reference/syntax/index.html, the menu itself. The Comments section was the one feature on that page shown as a dead code block while its fourteen neighbours were live examples, which is what a feature that used to throw looks like in a doc.
  3. docs/content/examples/percussion/index.html and docs/content/examples/organizing-compositions/index.html, the two pages with sections called "Using Comments" that used \line() titles instead of comments.
  4. docs/content/examples/ragam-tanam-pallavi/index.html, docs/content/examples/tisram/index.html, docs/content/reference/embellishments/index.html, the rest of the rewrites.
  5. Everything else, mechanical.

How it works

The bulk of the rewrite was scripted, applied inside <notation> bodies only, and never to a line starting with >, which is a SINGLE_LINE_RAW_STRING where everything after the > is markdown passed through. An earlier version of the script missed that and turned > ### Pallavi into > "###" Pallavi, which is how the exclusion got there.

for each <notation> body, for each line that does not start with >:
    drop the line entirely if it starts with \accidentals or \noTala
    quote any whitespace-delimited token containing #      M#     -> "M#"
                                                           GM#P   -> "GM#P"
    drop standalone | and || tokens
    replace each _ in an underscore run with one comma      ___   -> , , ,
Construct Rewritten to What that costs
| and || removed Nothing. beatviews.ts:139 already draws a line at the end of each bar and a double line at the end of the avartanam, from the cycle
___, _ one comma each Duration is preserved, the commas are now visible where a silent space was meant
M#, GM#P "M#", "GM#P" Renders as written. Sharps carry no meaning to the layout engine either way
\accidentals(...) removed It declared nothing the engine used
\noTala removed The example falls back to the default cycle
((S R G) kampitam) [~ S R G] This one genuinely changes meaning, see below
[S R] ~:- [G M] [S R] [~:- G M] Nothing. Every embellishment attaches to the literal that follows it

Decision log

  • Rewrote the examples rather than extending the grammar. We do not know yet what the grammar should become, and getting the site rendering should not wait on that. The impact analysis for all seven constructs is in issue 27, with the questions each one raises.
  • Underscores became commas rather than being deleted. Deleting them would have changed the rhythm of every example that used them, which is a worse lie than a visible comma.
  • Sharps became quoted literals rather than being dropped. \line("Pallavi") in Kalyani without its M# is a different raga. A quoted literal renders the same text, so the examples still read correctly even though the engine treats it as an opaque label.
  • The four paren gamaka examples are the ones whose meaning changed. (R~G) in the old notation means one note ornamented toward another; [~ R G] is a group of two notes with kampitam on the first. They are not the same thing. Both note names survive and the example renders, which is the best available option until issue 27 is settled.
  • Two more paren gamaka examples in complex-gamakas were left alone because they sit inside an <!-- --> block and never reach the browser. The test strips commented regions for the same reason.
  • The test lives in docs/tests/ next to the existing playground-patterns.spec.ts rather than in src/tests/, since it tests the docs rather than the library.
  • Kept the _ row in the syntax reference table, marked "not implemented yet", rather than deleting it. It documents the intended language, and issue 17 is open against it.
  • stripComments scans the string instead of calling .replace(/<!--[\s\S]*?-->/g, ""). CodeQL flagged the replace as js/incomplete-multi-character-sanitization, high severity, and it was right that the result can still contain <!--. It was also wrong for that input in a way that mattered here: an unterminated comment swallows the rest of the document in a browser, and the regex version kept every example after it in the test set. Four assertions cover the edges, since the corpus has no unterminated comment today.

Risk / blast radius

  • Docs content only. No source file changes, so the library is untouched.
  • The test adds 297 cases to the suite, which is most of its case count and about two seconds of its runtime. It is describe.each over the examples, so a failure names the file and the example id.
  • Be paranoid about the raw markdown embeddings. Lines starting with > are passed through as markdown and were deliberately excluded from every rule, but <notation> bodies mix the two freely and a rule leaking into one of those lines is the failure mode that would not show up as a test failure. git diff docs/content | grep '^[+-]' | grep '^\s*[+-]>' comes back empty.
  • Be paranoid about the four paren gamaka rewrites, which are the only edits where someone who reads Carnatic notation might disagree with the result.
  • Related tickets: issue 27 carries the grammar analysis, issue 28 the unknown-command throw that two of these examples were hitting. Issues 19 and 17 are the two constructs from this set that were already filed.

Before / after

                                      before   after
live <notation> examples                 148     148
parse with no errors                     109     148
render an underscore as visible text       5       0

The Comments section of the syntax reference, which is the change most worth looking at:

-<p>Use <code>//</code> for line comments:</p>
-
-<pre><code>// This is a comment
-S R G M
-// Comments are ignored by the parser
-P D N S.
-</code></pre>
+<notation id="line-comments" showSource="true" caption="Line Comments">
+// This is a comment
+S R G M
+P D N S.   // and this one trails the notation
+</notation>

A representative example body:

  \line("Pallavi with Mridangam")
- Sw: S R G M , P D N S. , ___ ___ , ___ ___ |
- \line("Mridangam: ta ka dhi mi , ta ka jha nu , ___ ___ , ___ ___")
+ // mridangam: ta ka dhi mi , ta ka jha nu
+ Sw: S R G M , P D N S. , , , , , , , , , , , , , ,

Out of scope

  • The grammar itself. Issue 27 has the analysis per construct, issue 28 the unknown-command throw.
  • Guarding NotationBlock.updatePreview with a try/catch. A renderer should not trust a parser not to throw, but the parser is where that bug is, and it is in issue 28.
  • The visual test cases under docs/static/visual-tests/, which are a separate fixture set and all parse today.
  • The three js/html-constructed-from-input alerts CodeQL has open against NotationBlock.ts. They predate this branch, which is why the check was green before. Filed as issue 30, along with the unescaped source listing on line 89 that the alerts do not cover.

39 of the 148 live <notation> examples in docs/content did not parse.
NotationBlock.updatePreview renders nothing when load() reports errors, so
those were blank boxes on the site with the reason only in the browser
console. Five more parsed but contained ___, which lexes as an ordinary
identifier and renders as visible text where a silent space was meant.

The examples were written against a syntax the parser does not implement.
Rather than guess at the grammar changes, this rewrites them onto what works
today and files issue 27 with the impact analysis for each construct, to be
evaluated once the site renders.

Rewrites, applied inside <notation> bodies and never to a line starting with
>, which is a raw markdown embedding:

  | and ||    removed. beatviews.ts already draws a bar line at the end of
              each bar and a double line at the end of the avartanam, from
              the cycle, so these characters were redundant with what gets
              drawn anyway.
  ___ and _   one comma per underscore, which keeps the duration the example
              was written with at the cost of showing the commas.
  M#, GM#P    quoted, so the text renders as written. Sharps are excluded
              from IdentChar and carry no meaning to the layout engine either
              way.
  \accidentals, \noTala
              removed. Neither command exists, and both threw out of load()
              rather than reporting an error, see issue 28.

Four examples using the paren gamaka syntax, ((S R G) kampitam) and (R~G),
are rewritten with the ~ operators that exist. Those are the ones whose
meaning changed, since the two notations are not equivalent. Four more in
reference/embellishments had the operator in a position the grammar does not
allow, since every embellishment attaches to the literal that follows it.

Prose that described the removed constructs is corrected rather than left
claiming things that do not work.

The Comments section of the syntax reference was the one feature on that page
shown as a dead code block while its fourteen neighbours were live examples.
It is live now, as are the two example pages whose "Using Comments" sections
used \line() titles instead of comments.

docs/tests/examples.spec.ts asserts both properties for every example and
runs in CI with the rest of the suite.
Comment thread docs/tests/examples.spec.ts Fixed
CodeQL flagged the replace as js/incomplete-multi-character-sanitization,
high severity: a single pass over `<!--[\s\S]*?-->` leaves a dangling `<!--`
behind when a comment never closes, so the result can still contain the
opener it was meant to remove.

It is also wrong for that case. An unterminated comment swallows the rest of
the document in a browser, and the regex version kept every example after it
in the test set. Walking the string handles both, and four assertions cover
the edges the corpus does not currently contain.
@panyam
panyam merged commit 9e14f92 into master Sep 21, 2026
6 checks passed
@panyam
panyam deleted the fix/docs-examples-parse branch September 21, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants