feat(theme): site-to-theme assembly + blocks-engine npm packaging

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Report a problem with a package in this repository
+title: ''
+labels: bug
+assignees: ''
+---
+
+**Package and version**
+Which package (e.g. `@automattic/blocks-engine`) and what version?
+
+**What happened**
+A clear description of the bug, including any `BlocksEngineError` `code` and `hint` you saw.
+
+**Reproduction**
+Minimal steps or code/CLI command that triggers the problem. Include sample input if relevant.
+
+```
+# command or code
+```
+
+**Expected behavior**
+What you expected to happen instead.
+
+**Environment**
+- Node version:
+- OS:
+- Install method (npm/npx/pnpm):

.github/workflows/js.yml

@@ -0,0 +1,48 @@
+name: JS Transformer
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/js.yml"
+      - "packages/blocks-engine/**"
+  push:
+    branches:
+      - trunk
+    paths:
+      - ".github/workflows/js.yml"
+      - "packages/blocks-engine/**"
+
+jobs:
+  validate:
+    name: Typecheck, build, and test
+    runs-on: ubuntu-latest
+
+    defaults:
+      run:
+        working-directory: packages/blocks-engine
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          version: 9
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: Typecheck
+        run: pnpm typecheck
+
+      - name: Build
+        run: pnpm build
+
+      - name: Run tests
+        run: pnpm test

.gitignore

@@ -0,0 +1,5 @@
+.claude/worktrees/
+.superpowers/
+dist/
+docs/superpowers/
+node_modules/

CONTEXT.md

@@ -0,0 +1,47 @@
+# blocks-engine (JS engine)
+
+The deterministic engine for WordPress blocks, shipped as one pluggable package (`@automattic/blocks-engine`) with a React-free default entry and an opt-in in-process `/wp` entry. It owns two generic, consumer-agnostic capabilities — **HTML→blocks translation** and **deterministic theme assembly** (static directory → block theme) — and exposes extension seams; consumers (the first being the data-liberation-agent static-site importer) plug their own choices, including non-deterministic quality steps, into those seams. Sibling to the existing `php-transformer`, with which it does **not** target output parity.
+
+(Identity note: the engine's scope deliberately expanded from "translation only" to "translation + theme assembly" — see ADR 0004, which supersedes the translation-only framing of ADR 0001.)
+
+## Language
+
+**Engine**:
+The translation + theme-assembly core plus its extension seams. Owns generic logic; knows nothing about any one consumer.
+_Avoid_: importer (that's the consumer)
+
+**Theme assembly / Assembler**:
+The engine's multi-**stage** process that turns a static site directory into a block theme on disk (ingest → foundation → section-extract → reconstruct → chrome → assets → plan → assemble). Run by the `siteToTheme` convenience command or by composing the stages directly.
+_Avoid_: pipeline (reserved for the **consumer's** end-to-end run, e.g. DLA's). The engine has an *assembler*, the consumer has a *pipeline*.
+
+**Stage**:
+One isolated unit of the assembler with a frozen input→output contract (e.g. `foundation`, `sectionExtract`, `assemble`). Stages are public and composable; `siteToTheme` chains them.
+
+**Hook**:
+An optional async seam at a named stage (`onFoundation`/`onSection`/`onAssets`/`onRefine`) where a consumer injects a **non-deterministic** quality step (visual polish, asset triage, repair). Absent hook = deterministic identity (byte-identical to no hook).
+_Avoid_: plugin, middleware.
+
+**SectionSpec**:
+The engine's shared contract describing one visual section (structure + style facts) consumed by reconstruction. Produced deterministically two ways: the engine's browser-free **cheerio** `sectionExtract` (best-effort, from static HTML), or **injected** by a consumer that captured richer computed-style specs (e.g. DLA's Playwright `extractFull`). Injected via the `sections` data input — NOT a hook (it is data, not a non-deterministic step).
+
+**ThemeModel**:
+The pure in-memory result of assembly (`styleCss`, `themeJson`, `templates`, `parts`, `patterns`, `assets`). Materialized to disk by `writeTheme`. Keeps the assembler snapshot-testable without disk.
+
+**Converter**:
+A pluggable unit (`(html, context) → block markup | null`) the engine applies to translate an HTML fragment. Built-ins ship with the engine; consumers may supply their own.
+_Avoid_: recipe (a consumer's word for its platform-specific converters)
+
+**ConversionContext**:
+The context object threaded through conversion (e.g. source URL, media-URL map). Carries only generic fields — never consumer-specific state.
+_Avoid_: BlockRecipeContext (the old consumer-coupled name)
+
+**Fallback block**:
+The block the engine emits for a fragment it can't convert. Defaults to `core/html`; a consumer overrides it via `htmlFallback` (a block name, or an emitter function).
+_Avoid_: island (a consumer's term for its own fallback shape)
+
+**Canonicalize**:
+Normalizing block markup through `@wordpress/blocks` so WordPress's parser/validator accepts it on the way in. One of the two `@wordpress/blocks`-backed operations (with the rawHandler conversion); both are process-isolated because they pull in React.
+_Avoid_: fix, normalize (use "canonicalize")
+
+**Transformer**:
+The PHP sibling package (`php-transformer`) — a separate engine with its own `TransformerResult` contract. Referenced only to say the JS engine does not mirror it.

CONTRIBUTING.md

@@ -0,0 +1,34 @@
+# Contributing
+
+This repository is a monorepo of WordPress block tooling. Each package is developed and released independently:
+
+- `packages/blocks-engine` — `@automattic/blocks-engine` (JS/TS engine + CLI)
+- `php-transformer` — `automattic/blocks-engine-php-transformer` (Composer)
+- `figma-transformer`
+
+## Reporting bugs
+
+Open an issue using the bug-report template. Include the package affected, the version, a minimal reproduction, and what you expected versus what happened.
+
+## Developing `@automattic/blocks-engine`
+
+```sh
+cd packages/blocks-engine
+pnpm install        # install dependencies (uses the committed lockfile)
+pnpm build          # build dist via tsup
+pnpm test           # run the vitest suite
+```
+
+The worker pool forks Node child processes, so a development environment must support `child_process` fork + IPC. When running tests or tools against the built output, run `pnpm build` first so the worker child file is present in `dist`.
+
+## Pull requests
+
+- Branch off `trunk`.
+- Keep changes scoped to one package per PR where practical; CI runs per-package on paths under that package.
+- Use Conventional Commits (`feat`, `fix`, `docs`, `build`, `test`, `refactor`, `chore`) with a package scope, e.g. `fix(blocks-engine): …`.
+- Add or update tests for behavior changes. The suite must pass before review.
+- Public API and `BlocksEngineError` `code` values are a stable contract — call out any change to them in the PR description and the package `CHANGELOG.md`.
+
+## Releasing
+
+Releases are tagged per package (`blocks-engine-v*`, `php-transformer-v*`, `figma-transformer-v*`). See the package's own release notes/tooling for the current process.

README.md

@@ -4,5 +4,6 @@ Blocks Engine is a collection of tools for generating, transforming, and materia
 
 ## Packages
 
+- [`@automattic/blocks-engine`](packages/blocks-engine/) - JavaScript primitives for transforming content into WordPress-native block outputs.
 - [`php-transformer`](php-transformer/) - PHP primitives for converting HTML, Markdown, and generated website artifacts into WordPress-native block outputs.
 - [`figma-transformer`](figma-transformer/) - PHP primitives for converting Figma `.fig` archives and Figma-derived scenegraphs into static HTML website artifacts with parity diagnostics.

docs/adr/0001-engine-owns-translation-and-composition.md

@@ -0,0 +1,15 @@
+# blocks-engine owns HTML→blocks translation and composition behind a pluggable extension API
+
+blocks-engine owns the generic work of translating HTML into WordPress blocks — the `@wordpress/blocks` rawHandler core, the block-markup canonicalizer, the deterministic converter catalog, **and** the precedence that composes converters in order. Consumer-specific choices are not baked in; they are supplied through three extension seams: a `ConversionContext`, an ordered list of consumer **Converters** (functions or declarative `selector→block` recipe tables), and an `htmlFallback` override (default `core/html`). The data-liberation-agent is the first consumer but won't be the only one, so the boundary is: *generic translation logic + composition + extension seams → engine; the specific things plugged into those seams → consumer.*
+
+## Considered Options
+
+- **Engine = low-level primitives only; each consumer composes.** Rejected: duplicates the generic catalog and the recipe-table mechanism in every consumer, leaving "the hard work" outside the shared engine.
+- **Engine = generic conversion only; consumers own precedence/composition.** Rejected: a second consumer would have to bring its own recipe-table engine and precedence logic; the reusable extensible surface stays trapped in the first consumer.
+
+## Consequences
+
+- The first consumer's types (`AdapterBlocks` / `BlockRecipe` / `BlockRecipeContext`) become, or alias, the engine's `Converter` / recipe-rule / `ConversionContext` types; the consumer passes its platform recipes as **data**.
+- "Behavior-preserving / byte-identical" constrains engine **output**, not interface — the type and precedence relocation is deliberate; golden fixtures pin the emitted markup.
+- The `htmlFallback` seam must be threaded through the engine's **internal recursion**, not just applied at the outer boundary — otherwise nested fallbacks lose the consumer's chosen block.
+- The consumer's fallback marker (DLA's `PIPELINE_ISLAND_OPENER`) and its install-time gate (`validateReplicaInputs`) stay in the consumer, supplied via `htmlFallback`.

docs/adr/0002-composable-operations-and-worker-pool.md

@@ -0,0 +1,20 @@
+# Engine API surface: composable operations + a worker pool
+
+The engine exposes three **composable operations** rather than one monolithic `convert()`:
+
+- `compose(html, ctx, { converters, htmlFallback })` — **pure, synchronous**, React-free. Runs consumer converters (functions or recipe tables) → built-in generic catalog → `htmlFallback`. No `@wordpress/blocks`.
+- `rawConvert(html)` and `canonicalize(markup)` — **async**, backed by a package-managed **pool** of `fork()`ed worker processes (default `min(cores, 4)`, configurable, dies with the parent). These load `@wordpress/blocks`, isolated in the pool.
+
+A convenience `convert()` chains `rawConvert → compose → canonicalize` for consumers who just want "HTML in, blocks out."
+
+(Packaging of these operations into one package with entry points is ADR 0003.)
+
+## Considered Options
+
+- **One monolithic `convert()` owning rawHandler internally.** Rejected: it would force consumer **function** converters (e.g. DLA's imperative Squarespace walk) through the worker IPC boundary, where functions can't be serialized — breaking function-converters under isolation. It would also force the consumer's currently-synchronous composition path to become async.
+- **Single forked child instead of a pool.** Rejected: throughput regression versus the current `cluster(min(cores, 4))` on whole-site runs, which is exactly the workload the parallelism exists for.
+
+## Consequences
+
+- Consumer **function** converters run in the pure, in-process `compose`; only HTML/markup **strings** cross the worker boundary. Serialization is never an issue.
+- The first consumer (DLA) deletes its bespoke HTTP server + cluster client and consumes the package's worker pool; it keeps wiring the three ops at its own pipeline stages.

docs/adr/0003-single-package-with-entry-points.md

@@ -0,0 +1,16 @@
+# Single package with entry points (not a pure/wp package split)
+
+Ship the engine as **one** package — `@automattic/blocks-engine` — with entry points rather than two packages:
+
+- **default** (`.`) — React-free: `compose` + helpers + `createWorker()` (the pool client). Never loads `@wordpress/blocks` in the importing process.
+- **`/wp`** — opt-in, **in-process** sync `rawConvert` / `canonicalize` / `convert` that load `@wordpress/blocks` in the current process; this is also what the forked worker child runs.
+
+## Considered Options
+
+- **Two packages (`@automattic/blocks-engine` pure + `@automattic/blocks-engine-wp`).** Rejected. The React isolation consumers need comes from the worker **process**, not a package boundary, so the split doesn't earn it. Its only real benefit — letting a helpers-only consumer skip the `@wordpress/*` + jsdom install — serves a largely hypothetical consumer: the package exists to do HTML→blocks, which needs rawHandler, and the cheerio-based pure code isn't browser-ready, so there's no edge/browser story either. The split also puts the headline name on the weaker half (the pure layer alone is a best-effort converter, not the engine).
+
+## Consequences
+
+- One version, one publish pipeline; no cross-package version coupling.
+- React isolation is a property of the worker `fork()`, reachable from the single package's default entry — the importing process never loads React 18.
+- A consumer with no React conflict can `import … from '@automattic/blocks-engine/wp'` for a synchronous in-process engine.

docs/adr/0004-engine-owns-deterministic-theme-assembly.md

@@ -0,0 +1,43 @@
+# Engine identity expands to deterministic theme assembly
+
+The engine's scope expands from **HTML→blocks translation only** to **translation +
+deterministic theme assembly**: a static site directory → a block theme on disk
+(`theme.json`, templates, parts, patterns, `style.css`, carried/rewritten assets),
+exposed as a `siteToTheme` convenience command over composable, public **stages**.
+
+This supersedes the framing in ADR 0001 / `CONTEXT.md` that treated a
+pipeline/importer as exclusively the **consumer's** job and the engine as a pure
+translator. The engine still knows nothing about any one consumer; "theme assembly"
+is generic, consumer-agnostic work — but it IS more than translation.
+
+Non-deterministic quality (visual-parity polish, decorative-asset triage, repair) is
+explicitly NOT engine code. It enters through optional async **hooks**
+(`onFoundation`/`onSection`/`onAssets`/`onRefine`); absent hook = deterministic
+identity. The engine owns the deterministic skeleton; consumers inject judgment.
+
+## Considered Options
+
+- **Keep the engine translation-only; theme assembly stays in the consumer (DLA).**
+  Rejected: the deterministic theme-assembly logic is generic (not DLA-specific) and
+  other implementers want it; leaving it in DLA blocks reuse, which is the whole point.
+- **Make theme assembly a separate sibling package.** Rejected: it shares the
+  translation core, the worker pool, and the React-isolation machinery; a package
+  split re-introduces the cross-package version coupling ADR 0003 avoided.
+- **Engine does its own browser capture to reach DLA-grade fidelity.** Rejected:
+  violates the browser-free posture and ADR 0003. Instead, `SectionSpec` is a shared
+  input contract — the engine ships a browser-free cheerio extractor for a best-effort
+  static path, and consumers inject richer captured specs for full fidelity.
+
+## Consequences
+
+- `CONTEXT.md` gains: Theme assembly / Assembler, Stage, Hook, SectionSpec, ThemeModel.
+  "pipeline" stays reserved for the consumer's end-to-end run; the engine has an
+  *assembler*.
+- The browser-free **static-dir** path is a NEW, lower-fidelity capability with its own
+  golden fixtures. It is NOT held to DLA-output parity.
+- The migration definition-of-done for lifting DLA's logic is: **engine + DLA-injected
+  specs/aggregates/hooks == DLA-today, byte-identical** (proves the lift preserved
+  behavior). Static-path output is governed by golden fixtures, not DLA parity.
+- DLA eventually deletes its deterministic copies and imports the engine, wiring its AI
+  skills into the hooks and feeding its captured `SectionSpec`s via the `sections` input.
+  The Studio/WP install stays in DLA.