Repository navigation
refactor(tooling): migrate tests to TypeScript and simplify tooling - #1219
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67cd937de6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Reviewed the actual PR contribution at 4d0fec117854ee06adbf77c7050e6239523bf88b against 634532d7f7fb21ccca49b4bca9bb9fa1df9308d6 with three independent full-diff review passes and root synthesis, including every changed test and fixture. No actionable introduced issue was identified.
Inspected build/workflow/package/tsconfig changes, generated-output cleanup, new build scripts, module loading and source-reference helpers; synthesized all three full-diff reports covering the TypeScript migration and all changed tests. The cleanup in the MCP build removes generated tests/**/.js, but its scripts build does not delete stale scripts/.mjs after an .mts source is removed or renamed. That incremental-build limitation remains; the earlier blanket statement about generated JS/MJS cleanup was too broad. No current missing source or executed stale-script failure was established in this bounded follow-up.
Static source review only; tests and builds were not run locally. This approval does not establish whole-stack integration or deployment.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d0fec1178
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Merge main and remove redundant build paths, fixture scaffolding, and evaluation adapters while preserving supported entrypoints and coverage.
Reuse the temporary-directory registry and existing API fixtures, and remove private runner, scoring, and reducer fixture intermediates.
mldangelo-oai
changed the title
Preserve main's updated runtime, session fixtures, and dependency pins while retaining the native TypeScript tests and tooling cleanup.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The broad cross-platform migration changes CI, subprocess, test-runner, MCP, and evaluation execution boundaries and warrants final human review.
Review effort: Balanced
Findings: None
What changed in this PR
Migrates test and evaluation tooling to directly executed TypeScript, removes generated JavaScript paths, and updates CI and documentation accordingly.
Changes:
- Converts MCP, SDK, and evaluation tooling to native TypeScript execution.
- Consolidates fixtures, subprocess helpers, reporters, and evaluation contracts.
- Updates type-checking, CI commands, package scripts, and documentation.
| File | Description |
|---|---|
sdk/typescript/tsconfig.json |
Includes CI runner and import rewriting. |
sdk/typescript/tsconfig.ci.json |
Removes redundant report-script inclusion. |
sdk/typescript/tests-ts/workbench-windows-compatibility.test.ts |
Uses canonical temporary directories. |
sdk/typescript/tests-ts/workbench-patch.test.ts |
Uses canonical temporary directories. |
sdk/typescript/tests-ts/workbench-canonical-paths.test.ts |
Reuses shared API fixtures. |
sdk/typescript/tests-ts/test-shards.test.ts |
Runs TypeScript CI scripts directly. |
sdk/typescript/tests-ts/test-reports.test.ts |
Tests direct TypeScript report comparison. |
sdk/typescript/tests-ts/support/test-subprocess.ts |
Uses URL-based subprocess working directory. |
sdk/typescript/tests-ts/support/temporary-directories.ts |
Shares MCP temporary-directory helpers. |
sdk/typescript/tests-ts/support/shell.ts |
Adds subprocess-output helper. |
sdk/typescript/tests-ts/skeleton.test.ts |
Updates expected CI command. |
sdk/typescript/tests-ts/security-policy-helper.test.ts |
Inlines synchronous fixture cleanup. |
sdk/typescript/tests-ts/runtime.test.ts |
Reuses API fixture management. |
sdk/typescript/tests-ts/deep-scan-workbench.test.ts |
Canonicalizes temporary paths. |
sdk/typescript/tests-ts/build-plugin.test.ts |
Adopts boolean fixture option. |
sdk/typescript/tests-ts/auth.test.ts |
Simplifies authentication fixtures. |
sdk/typescript/TESTING.md |
Documents TypeScript runner commands. |
sdk/typescript/scripts/test-shards.mts |
Adds native TypeScript types. |
sdk/typescript/scripts/test-shards.d.mts |
Removes redundant declarations. |
sdk/typescript/scripts/run-ci-tests.mts |
Migrates CI runner to TypeScript. |
sdk/typescript/scripts/compare-test-reports.mts |
Simplifies typed report comparison. |
sdk/typescript/scripts/ci-test-durations.json |
Removes excluded test timings. |
sdk/typescript/package.json |
Adds eval build and formatting coverage. |
sdk/typescript/.gitignore |
Removes obsolete emitted-script ignore. |
plugins/codex-security/mcp-app/tsconfig.json |
Type-checks tests and scripts. |
plugins/codex-security/mcp-app/tests/test_reporter.ts |
Tests TypeScript reporter discovery. |
plugins/codex-security/mcp-app/tests/test_python_command.ts |
Imports typed source directly. |
plugins/codex-security/mcp-app/tests/test_deep_scan_templates.ts |
Simplifies template assertions. |
plugins/codex-security/mcp-app/tests/test_deep_reducer_paging_eval.ts |
Reuses paging fixture and JSON helpers. |
plugins/codex-security/mcp-app/tests/test_artifact_storage.ts |
Adds typed MCP tool results. |
plugins/codex-security/mcp-app/tests/test_artifact_storage_regressions.ts |
Uses shared typed module importer. |
plugins/codex-security/mcp-app/tests/test_artifact_foundation.ts |
Consolidates source imports and JSON reads. |
plugins/codex-security/mcp-app/tests/test_artifact_deep_reducer_pages.ts |
Adds typed paging fixtures. |
plugins/codex-security/mcp-app/tests/test_artifact_attack_path.ts |
Shares JSONL helpers and types. |
plugins/codex-security/mcp-app/tests/support/temporary-directories.ts |
Adds typed tracking and cleanup. |
plugins/codex-security/mcp-app/tests/support/temporary-directories.d.mts |
Removes redundant declarations. |
plugins/codex-security/mcp-app/tests/support/streams.ts |
Types server lifecycle helpers. |
plugins/codex-security/mcp-app/tests/support/source-references.ts |
Adds typed source-reference helper. |
plugins/codex-security/mcp-app/tests/support/source-references.mjs |
Removes JavaScript helper. |
plugins/codex-security/mcp-app/tests/support/reducer-paging/deep-reducer-paging-server.ts |
Types paging server instrumentation. |
plugins/codex-security/mcp-app/tests/support/reducer-paging/controlled-code-mode.ts |
Simplifies controlled transport fixture. |
plugins/codex-security/mcp-app/tests/support/json.ts |
Centralizes JSON test utilities. |
plugins/codex-security/mcp-app/tests/scan-draft-recovery-fixture.ts |
Types shared recovery fixtures. |
plugins/codex-security/mcp-app/tests/scan-draft-fixture.ts |
Types draft fixture builders. |
plugins/codex-security/mcp-app/tests/sandbox-state.ts |
Types sandbox fixture input. |
plugins/codex-security/mcp-app/tests/import-module.ts |
Types esbuild module loading. |
plugins/codex-security/mcp-app/tests/build-server.ts |
Types bundled-server builds. |
plugins/codex-security/mcp-app/tests/assertions.ts |
Types common assertions. |
plugins/codex-security/mcp-app/TESTING.md |
Documents TypeScript MCP tests. |
plugins/codex-security/mcp-app/scripts/test_reporter.mts |
Implements streaming TAP/JUnit reporter. |
plugins/codex-security/mcp-app/scripts/test_reporter.mjs |
Removes JavaScript reporter. |
plugins/codex-security/mcp-app/scripts/test_host_build.mts |
Migrates host-build checks to TypeScript. |
plugins/codex-security/mcp-app/package.json |
Runs MCP tests from TypeScript. |
evals/tsconfig.json |
Adds unified eval type-checking. |
evals/triage-finding/types.ts |
Defines shared evaluation contracts. |
evals/triage-finding/tests/ticket-intake.yaml |
References TypeScript assertion. |
evals/triage-finding/tests/invocation-behavior.yaml |
References TypeScript assertion. |
evals/triage-finding/tests/github-rest-intake.yaml |
References TypeScript assertion. |
evals/triage-finding/scripts/test-triage-io.mts |
Migrates triage assertion checks. |
evals/triage-finding/scripts/test-hydrate-calibration-repos.mts |
Migrates hydration checks. |
evals/triage-finding/scripts/test-hydrate-calibration-repos.js |
Removes JavaScript checks. |
evals/triage-finding/scripts/test-generate-calibration-tests.mts |
Migrates generator checks. |
evals/triage-finding/scripts/test-generate-calibration-tests.js |
Removes JavaScript checks. |
evals/triage-finding/scripts/test-calibration-evidence.mts |
Migrates evidence checks. |
evals/triage-finding/scripts/test-calibration-dataset.mts |
Migrates dataset validation. |
evals/triage-finding/scripts/test-calibration-dataset.js |
Removes JavaScript validation. |
evals/triage-finding/scripts/test-app-surface-instructions.mts |
Migrates instruction checks. |
evals/triage-finding/scripts/hydrate-calibration-repos.mts |
Migrates repository hydration. |
evals/triage-finding/scripts/hydrate-calibration-repos.js |
Removes JavaScript hydrator. |
evals/triage-finding/sastbench/scripts/test-sastbench-lib.mts |
Updates SastBench library checks. |
evals/triage-finding/sastbench/scripts/test-promptfoo-native-harness.mts |
Verifies native TypeScript harness. |
evals/triage-finding/sastbench/scripts/test-install-sastbench.mts |
Migrates installer checks. |
evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.mts |
Migrates hydration-plan checks. |
evals/triage-finding/sastbench/scripts/test-hydrate-sastbench-repos.js |
Removes JavaScript checks. |
evals/triage-finding/sastbench/scripts/sastbench-result.mts |
Adds typed outcome parsing. |
evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.mts |
Migrates Promptfoo launcher. |
evals/triage-finding/sastbench/scripts/run-sastbench-promptfoo.js |
Removes JavaScript launcher. |
evals/triage-finding/sastbench/scripts/install-sastbench.mts |
Migrates installer implementation. |
evals/triage-finding/sastbench/scripts/hydrate-sastbench-repos.js |
Removes JavaScript hydrator. |
evals/triage-finding/sastbench/scripts/generate-sastbench-tests.js |
Removes redundant adapter. |
evals/triage-finding/sastbench/promptfooconfig.sastbench.yaml |
References native TypeScript hooks. |
evals/triage-finding/sastbench/promptfooconfig.sastbench-sample.yaml |
Uses shared TypeScript generator. |
evals/triage-finding/sastbench/assertions/sastbench-verdict.mts |
Migrates verdict assertion. |
evals/triage-finding/sastbench/assertions/sastbench-metrics.mts |
Implements typed native metrics hook. |
evals/triage-finding/sastbench/assertions/sastbench-metrics.js |
Removes JavaScript metrics hook. |
evals/triage-finding/README.md |
Documents TypeScript eval tooling. |
evals/triage-finding/promptfooconfig.yaml |
References TypeScript assertion. |
evals/triage-finding/promptfooconfig.calibration.yaml |
References TypeScript assertions. |
evals/triage-finding/promptfooconfig.calibration-smoke.yaml |
References TypeScript assertions. |
evals/triage-finding/datasets/README.md |
Updates tooling filenames. |
evals/triage-finding/assertions/triage-io.mts |
Migrates triage I/O assertion. |
evals/triage-finding/assertions/ticket-intake.mts |
Migrates ticket-intake assertion. |
evals/triage-finding/assertions/ticket-intake.js |
Removes JavaScript assertion. |
evals/triage-finding/assertions/output.mts |
Centralizes typed output parsing. |
evals/triage-finding/assertions/output.js |
Removes JavaScript output helper. |
evals/triage-finding/assertions/missing-input.mts |
Migrates missing-input assertion. |
evals/triage-finding/assertions/github-rest-intake.js |
Removes JavaScript assertion. |
evals/triage-finding/assertions/calibration-evidence.mts |
Migrates calibration evidence assertion. |
evals/secret-discovery/runtime.mts |
Migrates isolated eval runtime. |
evals/secret-discovery/runtime.mjs |
Removes JavaScript runtime. |
evals/secret-discovery/run.mts |
Migrates eval entrypoint. |
evals/secret-discovery/README.md |
Documents TypeScript execution. |
evals/secret-discovery/harness.mts |
Types the eval harness. |
evals/secret-discovery/grade.mts |
Types grading contracts. |
evals/secret-discovery/fixtures.mts |
Types and simplifies fixtures. |
evals/deep-reducer/run.mts |
Migrates reducer eval entrypoint. |
evals/deep-reducer/README.md |
Updates reducer eval commands. |
.github/workflows/test-quality.yml |
Runs report comparison from TypeScript. |
.github/workflows/node-ci.yml |
Builds and tests TypeScript tooling. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
alandelong-oai
left a comment
alandelong-oai
left a comment
There was a problem hiding this comment.
Reviewed fb11a6f2d432b8df1288eb745ee1dc7e70bf3007 against 86275c46833b4b66f3f5305aa42c0cc202009506 with exactly three independent HIGH full-diff passes followed by root synthesis. All 144 changed paths and 29,252 diff lines were reviewed, including every changed test and fixture. No supported introduced actionable defect was identified.
The earlier stale generated test/eval/script concerns are addressed in the reviewed source by direct TypeScript execution and removal of the emitting pipeline. Inspected the changed workflow commands, package scripts, no-emit configurations, runner result handling and optional reporting behavior.
Static source review only: no product tests, builds or models were run. GitHub's head status rollup remains FAILURE. Run 37287121843 attempt 1 is cancelled, with four failed coverage/selected-check aggregates; the available metadata does not establish the cancelling actor or a product failure cause. This source approval does not establish passing CI, merge readiness, successful whole-stack/current-main integration or deployment.
Summary
Move the remaining MCP tests and non-bootstrap JavaScript tooling to TypeScript while removing redundant build paths and fixture machinery. The complete PR is 1,205 maintained lines smaller than its merged main baseline (
86275c4).Changes
Testing
On the final merged source with frozen-lock Codex/Codex SDK 0.162.0-alpha.12 dependencies installed in an isolated copy:
Earlier qualified cleanup also passed standalone plugin imports, an offline six-row native Promptfoo baseline/candidate comparison, and an independent metric negative control. Secret-discovery's prior 135-test result is historical; these results are not represented as fresh whole-suite checks of the final main merge.
Risk and rollout
The final PR changes tests and tooling, with shipping production source and dependency pins matching merged main. Native TypeScript paths replace emitted JavaScript tooling paths; Node 22.13 uses --experimental-strip-types. Counts include all maintained additions, tests, configuration and documentation after normal formatting. Generated output and file moves receive no savings. Native Windows and live model evaluations were not run locally; remote CI is reported separately. Further cleanup proposals were deferred to prioritize publishing the verified change.
Public disclosure review