Skip to content

feat: transpile a chunk of files under one shared ts.Program (createProgramBatch) - #63

Merged
frosty00 merged 2 commits into
ccxt:masterfrom
carlotestor:feat/create-program-batch
Aug 1, 2026
Merged

feat: transpile a chunk of files under one shared ts.Program (createProgramBatch)#63
frosty00 merged 2 commits into
ccxt:masterfrom
carlotestor:feat/create-program-batch

Conversation

@carlotestor

Copy link
Copy Markdown

What

Adds Transpiler.createProgramBatch(paths) — transpile a set of files under one shared ts.Program instead of one program per file.

const batch = transpiler.createProgramBatch(files);
for (const f of files) {
    out[f] = batch.transpileGoByPath(f);   // same names as transpiler.transpileGoByPath
}

TranspileProgramBatch mirrors every transpile*ByPath name, so migrating a loop is literally transpiler.batch.. All six languages come from one generic transpileByPath(lang, path). No existing signature is touched; src/types.ts is unchanged.

Why

Every transpile*ByPath builds a program over [file, globalsShimPath]. The ITranspileProgramCache from #60 already makes the ~340-file import closure parse-free across calls, but the bind/check work behind ts.getPreEmitDiagnostics is redone for every file — and that is the dominant cost when a consumer transpiles a whole directory. Batching pays it once per chunk.

Measured on ccxt

104 exchanges, --force --rest-and-ws, idle 8-core box, chunk = 26 files per worker task:

lang 4 workers, 1 file/task 4 workers, batched 1 worker, 1 file/task 1 worker, batched
java 17 796 ms 14 455 ms (−19%) 33 484 ms 21 626 ms (−35%)
go 32 279 ms 28 110 ms (−13%) 41 547 ms 29 291 ms (−30%)
csharp 16 109 ms 12 402 ms (−23%) 34 276 ms 19 804 ms (−42%)

Single-thread microbenchmark on 8 exchange files (parents + derived, real ccxt classNameMap config): 1.85–1.95× faster, 8/8 outputs identical.

Correctness

The generated tree is byte-identical: md5 over the whole emitted java/ (797 files), go/ (713) and cs/ (1432) tree is unchanged versus the unbatched path, across every worker/chunk combination measured — including versus a pristine run with the consumer-side change stashed out.

Files that import each other (a derived exchange and its parent) are fine in one batch: they are separate root files of the same program, exactly as tsc would compile a project.

Design notes

  • The batch deliberately does not become cache.byPathOldProgram. An N-root program never structurally reuses a program built from a different root set, and holding the previous chunk's checker alive while building the next doubles peak RSS — the opposite of why a caller chunks. The cross-batch saving comes from the shared compiler host + sourceFiles map, which stay in the cache (measured: chunk A parsed 337 SourceFiles, chunk B added only 7).
  • Error isolation is free — the context is rebuilt from the program on every call, so a caller can try/catch per file and keep going. A nonexistent path throws out of its own call and leaves the batch usable.
  • checkFileDiagnostics inside setContextForPath is load-bearing, not optional: the printers read checker state back from it.
  • The batch borrows the owning Transpiler's printers/context, so don't drive that same Transpiler through another path mid-loop — use cloneSharingProgramCache() for a second independent driver.

Tests

tests/createProgramBatch.test.ts (4 new): batch output equals per-file output, all roots live in one program, a non-root path throws without poisoning the batch, and the batch does not clobber byPathOldProgram (single-file transpiles still work afterwards).

381/381 passing (377 existing + 4 new).

The dist/ commit

Kept as a separate commit so you can drop it and regenerate. It is here because package.json has no prepare/prepack script and the dist-regenerating CI step is commented out in .github/workflows/node.js.yml, while consumers install this as github:ccxt/ast-transpiler#master — so a src-only PR would never reach them.

The committed dist/ was exactly in sync with src/ before this (a pristine rebuild of the parent commit reproduced it byte-for-byte), so the whole dist diff is createProgramBatch and nothing else — no unrelated drift folded in.

Verified consumer-side: npm install github:carlotestor/ast-transpiler#33b848f into a scratch dir exposes createProgramBatch on the installed artifact.

Every transpile*ByPath call builds its own ts.Program over
[file, globalsShimPath]. The ITranspileProgramCache already makes the
~340-file import closure parse-free across calls, but the bind/check
work behind ts.getPreEmitDiagnostics is redone for every file, and that
is the dominant cost when a consumer transpiles a whole directory.

createProgramBatch(paths) compiles the paths as the root files of ONE
program and returns a session object mirroring the transpile*ByPath
names, so a caller migrates by replacing

    for (const f of files) transpiler.transpileGoByPath(f)

with

    const batch = transpiler.createProgramBatch(files);
    for (const f of files) batch.transpileGoByPath(f)

Files that import each other (a derived class and its parent) are fine
in one batch: they are separate root files of the same program, exactly
as tsc would compile a project. The emitted content, imports, exports
and methodsTypes are identical to the per-file path.

The batch deliberately does not become the cache's byPathOldProgram: an
N-root program never structurally reuses a program built from a
different root set, and keeping the previous chunk's checker alive while
the next is built would double peak memory, which is the opposite of why
a caller chunks. The cross-batch saving comes from the shared compiler
host and SourceFile cache, which stay in the ITranspileProgramCache.

Error isolation is free because the context is rebuilt from the program
on every call, so a caller can try/catch per file and keep going.

Measured on ccxt (104 exchanges, REST+WS, 4 worker threads, chunk 26):
java 17.8s -> 14.5s, go 32.5s -> 28.2s, csharp 16.1s -> 12.6s, with the
generated trees byte-identical (md5 over the whole tree unchanged).
The committed dist/ is the shipped artifact: package.json has no
prepare/prepack script and the CI step that regenerated dist is
commented out in .github/workflows/node.js.yml, while consumers install
this as github:ccxt/ast-transpiler#master. A src-only change would
therefore never reach them.

The committed dist was exactly in sync with src before this: a pristine
rebuild of the parent commit reproduced it byte-for-byte, so the whole
diff here (+68 lines) is createProgramBatch and nothing else.

Kept as a separate commit so it can be dropped and regenerated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants