Skip to content

develop.md

Latest commit

 

History

History
199 lines (169 loc) · 11.8 KB

develop.md

File metadata and controls

199 lines (169 loc) · 11.8 KB
Raw

Development Guide

This guide is for developers contributing to the wasmtex codebase.

Before modifying engine internals, build flags, memory handling, or execution glue for performance, read the engine optimization policy. It defines standalone compatibility evidence and the separate integrator adoption boundary.

Quick Start

npm install
npm run sync-engine-assets -- --from https://corca-ai.github.io/wasmtex/
npm run dev               # Start dev server
# App: http://localhost:6001

Prerequisites

  • Node.js: v24+ (engines.node: ">=24").
  • WASM Assets: Each engine's authored controller (*.worker.js), generated module (*.js), binary (*.wasm), and format (*.fmt/*.fmt.gz) must be present. The runtime and CI load them from a per-year subdirectory, public/wasmtex/<version>/ (e.g. public/wasmtex/2025/). Use npm run sync-engine-assets to fetch a hash-verified set. See docs/engine.md.

Commands

Command Description
npm run dev Vite dev server (port 6001)
npm run build Production build: typecheck (tsgo) + standalone demo app (vite build) → dist/ (GitHub Pages; gitignored)
npm run build:lib SDK-only build (BUILD_MODE=lib): the seven ES entry points (wasmtex, headless, node, synctex, lsp, lsp-monaco, lsp-server) + wasmtex.css → lib/ (committed)
npm run check Typecheck only (tsgo --noEmit)
npm run test Unit tests (Vitest, vitest run)
npm run test:watch Unit tests in watch mode
npm run test:e2e End-to-end tests (Playwright)
npm run test:golden / npm run update:golden Golden-output tests (write/refresh e2e/goldens/*.json)
npm run lint / npm run lint:fix Lint (Biome) — check / apply fixes
npm run format Format code (Biome)
npm run gen:texlive-catalog -- --manifest <manifest> --output <dir> Generate immutable completion shards from a final texlive-provenance.json inventory
npm run check:texlive-catalog -- <manifest> <dir> Verify exact catalog coverage, deterministic bytes, hashes, and provenance
npm run check:deployed-completion -- --manifest <manifest> --base-url <url> Stream and hash every catalog/semantic source against the deployed TeX Live endpoint
npm run reconcile:deployed-completion -- --manifest <manifest> --mirror-root <dir> --base-url <url> --policy <json> Apply only reviewed, hash-pinned CDN absences/hotfixes before immutable catalog generation
npm run gen:tex-semantic-catalog -- --manifest <manifest> --mirror-root <root> --overrides <json> --output <dir> Extract and merge versioned class/package semantic shards, including exact option-gated color definitions, plus a coverage report
npm run check:tex-semantic-catalog -- --manifest <manifest> --mirror-root <root> --overrides <json> --catalog <dir> Regenerate and reject semantic schema, provenance, source-byte, or golden drift
npm run probe:tex-semantics -- --input <json> --command <probe> --output <json> Run an exact-profile probe with fail-closed OS network isolation and bounded time/memory
npm run test:license-tools Test provenance, catalog, release, and licensing scripts with Node's test runner
npm run sync-engine-assets -- --from <baseUrl> Download and SHA-256-verify a complete versioned engine set into public/wasmtex/<version>/
npm run compat Compatibility harness — compile a corpus and bucket failures (node scripts/compat/run.mjs; writes compat/report.{json,md})
node scripts/gen-bloom-filter.mjs Generate bloom data from the configured object store, or from TEXLIVE_MIRROR_ROOT for an unpublished release

Set WASMTEX_SMOKE_TEXLIVE_URL to an exact immutable snapshot URL when the opt-in Node, cross-host, or incremental smoke suites qualify a new mirror.

The committed lib/ bundle

WasmTex isn't on npm, so consumers npm install github:corca-ai/wasmtex#main. A github: install must yield a usable package without running a build (the prepare build step is skipped or blocked by some package managers, and would otherwise leave exports pointing at nonexistent files). So the published library bundle in lib/ is committed — built by npm run build:lib. The demo-app / GitHub-Pages build stays in the gitignored dist/.

If you change src/ in a way that affects the built output, run npm run build:lib and commit lib/ in the same change. The lib-fresh CI job rebuilds and fails the PR if lib/ drifts from src/. The build is deterministic (no sourcemaps / absolute paths; Node 24 everywhere), so a clean rebuild is byte-identical. lib/** is marked linguist-generated in .gitattributes, so it's collapsed in PR diffs.

Architecture & Internals

Testing

Unit Tests

We use Vitest. Tests are located in *.test.ts files alongside the source code.

npm run test

Tests must assert observable behavior. In particular, worker/controller changes should be exercised through protocol responses, an engine adapter, or a rebuilt engine smoke test; reading implementation files and checking that source strings occur in some order does not prove the feature works and is not an acceptable regression test.

src/lsp/__tests__/completion-performance.test.ts enforces the semantic-index budget on a 600-file active graph: indexing under 3,000 ms, warmed completion under 150 ms, a single file update under 100 ms, and retained semantic metadata under 8 MiB. These are CI guardrails, not end-user timing claims; change a threshold only with benchmark evidence in the PR. Runtime completion snapshot tests separately enforce engine/host record ceilings, a 2 MiB serialized retention ceiling, revision/profile rejection, stale-on-edit behavior, and output-neutral worker response mapping. Engine rebuild PRs must also run the Node smoke and cross-host parity gates below so the authored C/controller hook is exercised, not merely its TypeScript consumer.

E2E Tests

We use Playwright. These verify the full compilation loop, SyncTeX, and BibTeX integration.

# Playwright starts the dev server itself (reuses one already on port 6001)
npm run test:e2e

Cross-Host (Node) Engine Tests

The same from-source WASM engine runs under Node via installNodeWorkerHost (src/engine/node-host.ts, exported from wasmtex/node). The verification tests are env-gated so they stay out of the default npm run test; they read the engine assets from public/, so run npm run sync-engine-assets -- --from <baseUrl> first. To verify a local rebuild without replacing release artifacts, set WASMTEX_SMOKE_PUBLIC_DIR to an equivalent directory tree containing the rebuilt files.

# Off-browser pdfTeX smoke
NODE_COMPILE_SMOKE=1 npx vitest run src/engine/node-compile.smoke.test.ts

# Client/server parity vs the browser golden — pdflatex + lualatex + xelatex + bibtex
CROSS_HOST_PARITY=1 npx vitest run src/engine/cross-host-parity.smoke.test.ts

For the Unicode initialization-heap optimization, stage baseline and candidate asset trees separately, retaining identical WASM, generated JS, and base formats. worker-heap-restore.test.ts exercises authored controllers through compile messages, including zero suffixes, memory growth, and ICU re-snapshotting. The opt-in real-engine differential checks PDFs, auxiliary files, diagnostics, and repeated body/preamble edits, and rejects fallback format generation. It stages temporary controller copies with an identical fixed clock on both sides: dvipdfmx can embed dates inside compressed PDF objects, which the ordinary PDF date-field normalization does not remove. Release files are untouched.

WASMTEX_HEAP_BASELINE_DIR=/path/to/baseline-public \
WASMTEX_SMOKE_PUBLIC_DIR=/path/to/candidate-public \
WASMTEX_SMOKE_TEXLIVE_VERSION=2026 \
WASMTEX_SMOKE_TEXLIVE_URL=https://texlive.corca.ai/snapshots/2026-ba38749b8714505a/2026/ \
npx vitest run src/engine/unicode-heap-restore.smoke.test.ts

Repeat with the matching 2025 profile when both annual lines are affected. For an engine binary or build-flag change, also set WASMTEX_HEAP_REBUILT_ENGINE=1. This permits different WASM/generated JS while still requiring byte-identical compressed formats and rejecting format regeneration. Record how each candidate was built; this flag does not verify source provenance or qualify a binary edited outside the source build pipeline. This focused differential complements the browser and release gates in the engine optimization policy; it does not itself qualify a CorTeX successor.

For the broader Unicode feature comparison, use the same baseline/candidate and annual mirror variables with WASMTEX_UNICODE_COMPAT=1, then run npx vitest run src/engine/unicode-compatibility.smoke.test.ts. It reuses the baseline compressed formats and compares repeated Unicode math, PDF import, TikZ, project-local and Korean fonts, BibTeX, index, TeX error, nested main files, and recovery runs in the same worker. PDF bytes, available SyncTeX, geometry, diagnostics, recorder inputs, dependency graphs, glyph coverage, and bibliography/index outputs must match. This remains a focused corpus; source build and CorTeX rollout gates are separate requirements. The nested XeTeX case explicitly records an existing dvipdfmx output-path defect: after a previous successful root compile, it returns that previous PDF. Matching this known failure is evidence of unchanged behavior, not successful nested-path support. A separate correctness fix must update that expectation and qualify its intended output change. The fixtures are shared in e2e/unicode-compatibility-corpus.ts for browser differential runs.

Stage downloaded source builds for these tests without substituting their newly generated formats:

node scripts/stage-unicode-qualification.mjs \
  --year 2026 --source <full-candidate-commit> \
  --baseline /path/to/baseline-public/wasmtex/2026 \
  --xetex /path/to/downloaded-xetex-artifact \
  --luahbtex /path/to/downloaded-luatex-artifact \
  --output-public /path/to/new-qualification-public

The stager verifies receipts and artifact hashes, the source pin, toolchain, mirror, and expected candidate commit. It keeps the baseline formats, records both generation histories in QUALIFICATION-INPUTS.json, and removes release metadata that would describe the wrong bytes. Its output is test-only, not a release assembly or a substitute for corresponding-source qualification. One family option may be omitted to verify an earlier completed build; the other family then retains its baseline assets and receipt. Select the rebuilt engine's tests explicitly and do not count the unchanged family as a qualified rebuild.

Compile pipeline probe

e2e/compile-pipeline-probe.ts exports measureCompilePipeline(options) for a standalone development page. Supply explicit engine assets, mirror and a text main.tex; it runs cold, unchanged, body-edit, preamble-edit and restore cases through the real headless SDK. Run probes serially in one JS context. Worker command round-trips include synchronous I/O and message overhead. File writes and parent fetch-header timings can overlap and must not be summed into wall time. Unsupported worker CPU/network timings remain unobserved rather than being reported as zero. Integrators measure their own debounce, queue and viewer paint separately; this helper imports no application code and is not shipped in the SDK bundle. Use a fixed test clock for PDF hash comparisons and record both preload and initialization cost when evaluating a warmup candidate.