feat: transpile a chunk of files under one shared ts.Program (createProgramBatch) - #63
Merged
Merged
Conversation
Every transpile*ByPath call builds its own ts.Program over
[file, globalsShimPath]. The ITranspileProgramCache already makes the
~340-file import closure parse-free across calls, but the bind/check
work behind ts.getPreEmitDiagnostics is redone for every file, and that
is the dominant cost when a consumer transpiles a whole directory.
createProgramBatch(paths) compiles the paths as the root files of ONE
program and returns a session object mirroring the transpile*ByPath
names, so a caller migrates by replacing
for (const f of files) transpiler.transpileGoByPath(f)
with
const batch = transpiler.createProgramBatch(files);
for (const f of files) batch.transpileGoByPath(f)
Files that import each other (a derived class and its parent) are fine
in one batch: they are separate root files of the same program, exactly
as tsc would compile a project. The emitted content, imports, exports
and methodsTypes are identical to the per-file path.
The batch deliberately does not become the cache's byPathOldProgram: an
N-root program never structurally reuses a program built from a
different root set, and keeping the previous chunk's checker alive while
the next is built would double peak memory, which is the opposite of why
a caller chunks. The cross-batch saving comes from the shared compiler
host and SourceFile cache, which stay in the ITranspileProgramCache.
Error isolation is free because the context is rebuilt from the program
on every call, so a caller can try/catch per file and keep going.
Measured on ccxt (104 exchanges, REST+WS, 4 worker threads, chunk 26):
java 17.8s -> 14.5s, go 32.5s -> 28.2s, csharp 16.1s -> 12.6s, with the
generated trees byte-identical (md5 over the whole tree unchanged).
The committed dist/ is the shipped artifact: package.json has no prepare/prepack script and the CI step that regenerated dist is commented out in .github/workflows/node.js.yml, while consumers install this as github:ccxt/ast-transpiler#master. A src-only change would therefore never reach them. The committed dist was exactly in sync with src before this: a pristine rebuild of the parent commit reproduced it byte-for-byte, so the whole diff here (+68 lines) is createProgramBatch and nothing else. Kept as a separate commit so it can be dropped and regenerated.
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
Transpiler.createProgramBatch(paths)— transpile a set of files under one sharedts.Programinstead of one program per file.TranspileProgramBatchmirrors everytranspile*ByPathname, so migrating a loop is literallytranspiler.→batch.. All six languages come from one generictranspileByPath(lang, path). No existing signature is touched;src/types.tsis unchanged.Why
Every
transpile*ByPathbuilds a program over[file, globalsShimPath]. TheITranspileProgramCachefrom #60 already makes the ~340-file import closure parse-free across calls, but the bind/check work behindts.getPreEmitDiagnosticsis redone for every file — and that is the dominant cost when a consumer transpiles a whole directory. Batching pays it once per chunk.Measured on ccxt
104 exchanges,
--force --rest-and-ws, idle 8-core box, chunk = 26 files per worker task:Single-thread microbenchmark on 8 exchange files (parents + derived, real ccxt
classNameMapconfig): 1.85–1.95× faster,8/8outputs identical.Correctness
The generated tree is byte-identical: md5 over the whole emitted
java/(797 files),go/(713) andcs/(1432) tree is unchanged versus the unbatched path, across every worker/chunk combination measured — including versus a pristine run with the consumer-side change stashed out.Files that import each other (a derived exchange and its parent) are fine in one batch: they are separate root files of the same program, exactly as
tscwould compile a project.Design notes
cache.byPathOldProgram. An N-root program never structurally reuses a program built from a different root set, and holding the previous chunk's checker alive while building the next doubles peak RSS — the opposite of why a caller chunks. The cross-batch saving comes from the shared compiler host +sourceFilesmap, which stay in the cache (measured: chunk A parsed 337 SourceFiles, chunk B added only 7).try/catchper file and keep going. A nonexistent path throws out of its own call and leaves the batch usable.checkFileDiagnosticsinsidesetContextForPathis load-bearing, not optional: the printers read checker state back from it.Transpiler's printers/context, so don't drive that sameTranspilerthrough another path mid-loop — usecloneSharingProgramCache()for a second independent driver.Tests
tests/createProgramBatch.test.ts(4 new): batch output equals per-file output, all roots live in one program, a non-root path throws without poisoning the batch, and the batch does not clobberbyPathOldProgram(single-file transpiles still work afterwards).381/381 passing (377 existing + 4 new).
The
dist/commitKept as a separate commit so you can drop it and regenerate. It is here because
package.jsonhas noprepare/prepackscript and the dist-regenerating CI step is commented out in.github/workflows/node.js.yml, while consumers install this asgithub:ccxt/ast-transpiler#master— so a src-only PR would never reach them.The committed
dist/was exactly in sync withsrc/before this (a pristine rebuild of the parent commit reproduced it byte-for-byte), so the whole dist diff iscreateProgramBatchand nothing else — no unrelated drift folded in.Verified consumer-side:
npm install github:carlotestor/ast-transpiler#33b848finto a scratch dir exposescreateProgramBatchon the installed artifact.