Skip to content

refactor: replace process globals with an explicit transpile context - #60

Merged
frosty00 merged 3 commits into
ccxt:masterfrom
carlotestor:refactor/transpile-context-no-global
Jul 31, 2026
Merged

refactor: replace process globals with an explicit transpile context#60
frosty00 merged 3 commits into
ccxt:masterfrom
carlotestor:refactor/transpile-context-no-global

Conversation

@carlotestor

@carlotestor carlotestor commented Jul 31, 2026

Copy link
Copy Markdown

Problem

The typescript SourceFile, TypeChecker and Program are stored on the node global object:

global.src = sourceFile;
global.checker = typeChecker;
global.program = program;

and read back from ~96 call sites across baseTranspiler.ts and the six language printers (global.checker.getTypeAtLocation(...), global.src.getFullText(), ...).

Because that state is process-wide, two Transpiler instances cannot be used at the same time. The second createProgram* call overwrites the state the first one is still printing against, so the first transpilation silently resolves types against the wrong source file. That also rules out running several transpilations concurrently in one process, which is the natural way to speed up large multi-language builds.

Change

The state now lives in an explicit per-transpile context instead:

interface ITranspileContext {
    src: ts.SourceFile;
    checker: ts.TypeChecker;
    program: ts.Program;
}
  • Transpiler owns the context of the transpilation in flight and hands it to every language printer via setContext.
  • BaseTranspiler holds it in this.context and exposes getSrc() / getChecker() / getProgram(). All ~96 global.* reads became this.getChecker() / this.getSrc() / this.getProgram(), so each printer reads its own instance state.
  • createProgramInMemoryAndSetGlobals / createProgramByPathAndSetGlobals become ...AndSetContext and return the context they built. The old names stay as thin deprecated forwarders so no external caller breaks.
  • transpile(...)'s setGlobals parameter is renamed to createContext (same position, same semantics).
  • Reading the context before a program exists now throws a clear error instead of dereferencing whatever the previous transpilation left behind.

After this change global.src, global.checker and global.program are never written or read — a Transpiler instance is self-contained.

Compatibility

Public API is unchanged. transpileGoByPath, transpilePython, transpileDifferentLanguagesByPath, getFileImports, getFileExports and friends keep their signatures, so ccxt and other consumers need no call-site changes.

Verification

  • npm test369 passed, 9 suites (364 pre-existing + 5 new)
  • npm run build (lint + tsup) — passes, 0 eslint errors
  • npx tsc --noEmit — clean

Output is byte-for-byte identical. I transpiled tests/integration/source/transpilable.ts on master and on this branch and diffed the results: all 6 languages, both ByPath and ByContent modes, plus getFileImports / getFileExports / methodsTypes — no differences.

$ diff -r base new && echo IDENTICAL
IDENTICAL

npm run integration was also run; it only fails locally at the rust step because cargo is not installed in this environment (code: 127), unrelated to this change.

New tests/transpileContext.test.ts covers what the refactor buys:

  • global.src / global.checker / global.program stay undefined after a transpile
  • two Transpiler instances that interleave program creation keep their own context and both produce correct output (this is the case that was broken before)
  • every language printer sees the context of its owning transpiler
  • using a printer with no context throws instead of reading stale state

Sharing a program between threads

The follow-up question was whether a ts.Program can be shared between threads. It cannot, and that is a hard limit rather than a missing feature. Each worker thread is a separate V8 isolate and postMessage uses structured clone. Measured against typescript directly:

postMessage(ts.Program):             FAILED -> DataCloneError: () => rootNames could not be cloned
postMessage(ts.TypeChecker):         FAILED -> DataCloneError: ... could not be cloned
postMessage(ts.SourceFile):          FAILED -> DataCloneError: ... could not be cloned
postMessage(SourceFile.statements[0]): FAILED -> DataCloneError: ... could not be cloned

A SharedArrayBuffer only shares raw bytes, so it cannot hold a Program either. Anything that claims to share one across isolates is really rebuilding it on the other side.

What can be shared is the work behind the program, so this PR now does that.

ITranspileProgramCache

The parsed SourceFiles (the es lib chain plus the import closure seen so far) and the last program built from them move out of the Transpiler instance into a standalone cache:

const cache = Transpiler.createProgramCache();

const a = new Transpiler(config, cache);
const b = new Transpiler(config, cache); // reuses everything a has parsed
const c = a.cloneSharingProgramCache();  // same, from an existing instance
  • the constructor's second argument is optional — omitting it keeps the current behaviour of a private cache per instance, so nothing changes for existing callers
  • getProgramCache() / cloneSharingProgramCache() point further instances at an existing cache
  • the in-memory path uses the cache too, so repeated transpileX(content) calls no longer re-parse the lib chain every time

Each instance keeps its own printers and its own transpile context, so instances over one cache can be interleaved without clobbering each other — that is exactly what the context refactor above makes safe.

For worker pools

The cache is per thread, not per process: each worker keeps its own long-lived cache and the pool ships file paths as it does today. worker.ts now holds a module-level cache, so parse work happens once per worker instead of once per task.

On 30 ccxt source files in that shape (a fresh Transpiler per 5-file task, as a Piscina worker does):

per-task instance (today): 4528 ms  (150.9 ms/file)
shared program cache     : 2881 ms  ( 96.0 ms/file)   1.57x

Verification

  • npm test375 passed, 10 suites (369 + 6 new in tests/sharedProgramCache.test.ts)
  • output is unchanged: transpiled with isolated instances vs. a shared cache vs. interleaved clones, across all six languages, both modes, plus getFileImports / getFileExports / methodsTypes — byte for byte identical
  • typescript's oldProgram reuse does not invalidate a live program: a checker still resolves its own types after another instance built a program from the same cache (covered by a test)

Caches are same-thread only, and that is documented in the README next to the worker example.

The typescript SourceFile, TypeChecker and Program were stored on the node
`global` object (`global.src`, `global.checker`, `global.program`) and read
back from ~96 call sites across the language printers. Because that state is
process-wide, two Transpiler instances cannot be used at the same time: the
second `createProgram*` call overwrites the state the first one is still
printing against, so the output silently belongs to the wrong source file.

Hold the state in an `ITranspileContext` instead:

- `Transpiler` keeps the context of the transpilation in flight and hands it
  to every language printer through `setContext`
- `BaseTranspiler` exposes `getSrc()` / `getChecker()` / `getProgram()`, so
  the printers read their own instance state rather than process globals
- `createProgram*AndSetGlobals` become `createProgram*AndSetContext` and
  return the context they created; the old names remain as deprecated
  forwarders
- reading the context before a program exists now throws instead of
  dereferencing whatever the previous transpilation left behind

Public API is unchanged: `transpileGoByPath`, `transpile`, and friends keep
their signatures, and transpilation output is byte for byte identical.

Adds tests/transpileContext.test.ts covering instance isolation, printer
context propagation, and the absence of the globals.
Drop the getContext() helper and read the context field directly.
getSrc/getChecker/getProgram keep their existing behaviour.
@carlotestor
carlotestor force-pushed the refactor/transpile-context-no-global branch from 499cbc8 to 622731a Compare July 31, 2026 10:25
A ts.Program cannot be shared between worker threads: each worker is a
separate V8 isolate and postMessage uses structured clone, which rejects a
Program, a TypeChecker, a SourceFile and even a single AST node. A
SharedArrayBuffer only shares raw bytes, so it cannot hold one either.

What can be shared is the work behind the program. The parsed SourceFiles
(the es lib chain plus the import closure seen so far) and the last program
built from them move out of the Transpiler instance into a standalone
ITranspileProgramCache:

- `Transpiler.createProgramCache()` builds one, and the constructor takes it
  as an optional second argument; omitting it keeps the previous behaviour of
  a private cache per instance
- `getProgramCache()` / `cloneSharingProgramCache()` point further instances
  at an existing cache
- the in-memory path now uses the cache too, so repeated transpileX(content)
  calls no longer re-parse the lib chain every time

Each instance keeps its own printers and its own transpile context, so
instances over one cache can be interleaved without clobbering each other,
and output is unchanged. Verified byte for byte across all six languages,
both modes, plus imports/exports/methodsTypes.

For worker pools the cache is per worker: worker.ts now keeps a module-level
cache so parse work happens once per worker instead of once per task. On 30
ccxt source files in that shape (fresh instance per 5-file task) this is
4528ms -> 2881ms, 1.57x.

Adds tests/sharedProgramCache.test.ts covering cache sharing, context
independence, lib SourceFile reuse, output parity and concurrent use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants