Skip to content

test(core): fix late-suite harness regressions - #30

Closed
dzianisv wants to merge 1 commit into
mainfrom
issue-26-core-late-suite-failures
Closed

dzianisv wants to merge 1 commit into
mainfrom
issue-26-core-late-suite-failures

Conversation

@dzianisv

Copy link
Copy Markdown
Owner

Summary

  • keep the default injected core test model catalog additive instead of treating it as an authoritative custom catalog
  • add regression tests that distinguish the default test catalog path from explicit custom model catalogs
  • make stdio_server_bin() build codex-rmcp-client's test_stdio_server on demand under Cargo so fresh worktrees can run the RMCP-backed core suites

Root Cause

  • TestCodexBuilder was routing every config with a model_catalog through ThreadManager::new(...), which made the default injected test catalog authoritative and disabled remote /models refresh.
  • core integration tests that depend on test_stdio_server assumed the cross-package binary was already built, which is false in a fresh worktree running only codex-core tests.

Testing

  • cargo test -p core_test_support --lib -- --nocapture
  • cargo test -p codex-core --test all suite::remote_models::remote_models_truncation_policy_with_tool_output_override -- --nocapture --exact
  • cargo test -p codex-core --test all suite::search_tool::search_tool_selection_persists_across_turns -- --nocapture --exact
  • cargo test -p codex-core --test all suite::truncation::mcp_tool_call_output_exceeds_limit_truncated_for_model -- --nocapture --exact
  • cargo test -p codex-core --test all suite::rmcp_client::stdio_server_round_trip -- --nocapture --exact
  • cargo test -p codex-core --test all suite::sqlite_state::mcp_call_marks_thread_memory_mode_polluted_when_configured -- --nocapture --exact
  • cargo test -p codex-core --test all suite::search_tool:: -- --nocapture
  • cargo test -p codex-core --test all suite::rmcp_client:: -- --nocapture

Fixes #26

@dzianisv

Copy link
Copy Markdown
Owner Author

CI status after opening this PR:

  • The issue-#26 code changes are in and the local validation set passed.
  • PR #30 is currently blocked by the separate workflow issues already tracked as #27 and #28.

What the failing checks are showing right now:

  • build-test is failing inside scripts/stage_npm_packages.py because it cannot resolve a rust-release workflow for CODEX_VERSION=0.74.0.
  • The rust-ci matrix jobs are still failing before any steps run. The run metadata again shows empty steps arrays for those matrix entries.
  • The same runner/bootstrap pattern is also hitting the macOS Bazel matrix jobs.

I updated issue #26 with the actual two root causes fixed in this PR and the exact local validation set:

  • https://github.com/dzianisv/codex/issues/26#issuecomment-4072206406

Next follow-on work is still #27 / #28 so this PR can turn green and merge cleanly.

@dzianisv

Copy link
Copy Markdown
Owner Author

Superseded by #35, which now carries this branch's changes on the consolidated integration branch.

@dzianisv dzianisv closed this Mar 17, 2026
@dzianisv
dzianisv deleted the issue-26-core-late-suite-failures branch March 17, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix codex-core late-suite failures in remote model and search-tool flows

1 participant