Skip to content

Add streaming XML parsing with SAX2 callbacks - #171

Open
shunkica wants to merge 3 commits into
jameslan:masterfrom
shunkica:sax-push-parser
Open

shunkica wants to merge 3 commits into
jameslan:masterfrom
shunkica:sax-push-parser

Conversation

@shunkica

@shunkica shunkica commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this adds

A new XmlSaxParser for parsing large XML files without loading them entirely into memory. Instead of building a complete document tree like XmlDocument does, the parser processes chunks of raw bytes and fires callbacks for each piece it encounters — opening and closing tags, text content, comments, and so on.

This is useful when you're working with very large XML files and either don't need the whole document in memory at once, or only care about specific parts of it.

How it works

You create a parser with a set of optional callbacks, then feed it byte chunks:

import fs from 'node:fs';
import { XmlSaxParser } from 'libxml2-wasm';

const decoder = new TextDecoder();
let inTitle = false;
let title = '';

using parser = XmlSaxParser.create({
    startElementNs(localName) { inTitle = localName === 'title'; },
    endElementNs() { inTitle = false; },
    characters(data) {
        if (inTitle) title += decoder.decode(data, { stream: true });
    },
});

for await (const chunk of fs.createReadStream('doc.xml')) {
    parser.push(chunk);
}
parser.finish();

The parser handles the messy parts for you: chunks might split in the middle of a tag or even a UTF-8 character, but the parser reassembles them transparently. (One text node may still arrive as several characters calls, hence the streaming TextDecoder in the example.) Encoding is detected from the BOM or XML declaration, and callbacks always receive UTF-8 bytes.

Why this matters

Memory efficiency. A gigabyte XML file doesn't require a gigabyte of memory — the parser only keeps what it's actively working on in the buffer.

Pay for what you use. Callbacks are optional. If you don't care about processing instructions, you don't register that callback, and the events are skipped without even crossing the WebAssembly boundary. Large text nodes can be inspected as direct references to the parser's memory — valid only for the duration of the callback — rather than copied into JavaScript strings. And when you've seen the part of the document you care about, stop() ends the parse early without an error.

Clear error handling. If the XML becomes unparsable, push throws immediately. When finish returns without throwing, you know the whole document was well-formed — including namespace rules, which plain parseability alone doesn't cover. Non-fatal diagnostics are collected in a warnings getter instead of interrupting the parse.

Safe boundaries. Exceptions thrown in callbacks abort the parse cleanly rather than corrupting libxml2's internal state. Re-entering the parser from a callback is rejected with an error instead of silently breaking things.

What changed internally

  • Exposed libxml2's push-parser functions to the WASM binding
  • Refactored error construction so both SAX and DOM parsing can report problems consistently
  • Added comprehensive documentation (docs/sax.md) covering feeding strategies, encoding, entity quirks, and memory management

Compatibility notes

The parser rejects options that don't work in streaming mode (like DTD validation, which requires a complete tree) rather than silently misbehaving. Most standard parsing options are supported. The API is marked @alpha while the interface stabilizes based on feedback.

XmlSaxParser feeds the document to libxml2 as raw-byte chunks, split at
arbitrary positions, and reports its content through SAX2 callbacks
while it is being parsed, without building a document tree - peak
memory stays independent of the document size. The encoding is detected
from the document itself; callbacks always receive UTF-8.

The XmlSaxHandler callbacks cover document boundaries, namespace-aware
element events with declarations and attributes, character data and
CDATA (delivered as zero-copy views into the WebAssembly memory),
comments and processing instructions. All callbacks are optional:
events without a callback never cross the WebAssembly boundary.

Parsing can be ended early with stop(), including from within a
callback; push() and finish() throw XmlParseError on fatal
diagnostics, while warning-level ones are collected in `warnings`.
Handler exceptions abort the parse and are rethrown from push/finish
rather than unwinding through libxml2's C frames, and re-entering or
disposing the parser from a callback is rejected. Like the other
wrappers owning libxml2 memory, the parser is released with dispose()
or `using`, and the entity-store document libxml2 keeps for the
internal subset is freed with it, whether parsing completed, failed
or was stopped.

Adds the _xmlCreatePushParserCtxt, _xmlParseChunk, _xmlStopParser,
_xmlCtxtSetOptions and _xmlCtxtGetDocument exports to the binding.
The guard checked _malloc for the 0 it returns once the WebAssembly
heap cannot grow anymore, instead of writing the chunk to address 0.
But withCString is one of three marshalling sites that write to the
returned address unchecked - allocUTF8Buffer and allocCStringArray
make the same assumption - so guarding this one leaves the primitive
intact everywhere else, and none of it is SAX work.

Fixing all three belongs in its own change, together with the
try/finally that xmlCreatePushParserCtxt then needs: it frees its
xmlSAXHandler only after withStringUTF8 returns, so an allocUTF8Buffer
that can throw turns that into a leak.
Three branches in sax.mts cannot be reached from a test, leaving the
file short of the 100% the rest of src/ reports. Annotate them the way
the other defensive branches in the codebase already are:

- the null context from xmlCreatePushParserCtxt, which only happens
  when the WebAssembly heap cannot grow anymore;
- the empty details fallback, which needs an error storage index that
  a live context can never be missing - errorIndices is populated at
  creation and cleared only by the free that zeroes _ptr;
- the path out of parseChunk's catch into its finally, which V8 emits
  as a block and reports uncovered even though the catch always
  rethrows and can never take it.

sax.mts now reports 100% of statements, branches, functions and lines.
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Coverage after merging sax-push-parser into master will be

100.00%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
src
   c14n.mts100%100%100%100%
   diag.mts99.51%100%90.91%100%
   disposable.mts100%100%100%100%
   disposeShim.mts100%100%100%100%
   document.mts100%100%100%100%
   dtd.mts100%100%100%100%
   index.mts100%100%100%100%
   libxml2.mts100%100%100%100%
   metadataShim.mts100%100%100%100%
   nodejs.mts100%100%100%100%
   nodes.mts99.92%99.22%100%100%76
   sax.mts100%100%100%100%
   utils.mts100%100%100%100%
   validates.mts100%100%100%100%
   xpath.mts100%100%100%100%

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant