Skip to content

test(tokenless): run the hook script suites from test-integration - #3316

Open
Forrest-ly wants to merge 1 commit into
agentic-os-org:mainfrom
Forrest-ly:chore/tokenless-wire-hook-script-suites
Open

Forrest-ly wants to merge 1 commit into
agentic-os-org:mainfrom
Forrest-ly:chore/tokenless-wire-hook-script-suites

Conversation

@Forrest-ly

Copy link
Copy Markdown
Collaborator

What

make test-hooks has three members and no workflow calls it — its third member, tests/run-all-tests.sh, drives the release binary installed on PATH on purpose (see the note above its test 5.0b), so the target as a whole needs an installed tokenless and stays a developer-machine check. The other two members therefore ran nowhere but developer machines, and two contracts went unenforced in CI:

suite contract it pins
tests/test-run-hook-install-scope.sh run-hook.sh resolves a hook from its own adapter tree before the FHS and home-directory fallbacks, so concurrent RPM, Makefile and raw installs cannot cross-load another version
tests/test-tool-ready-readable-fixer.sh tool_ready_hook.sh keeps its hard bypass, so the dormant legacy four-phase flow — which runs any readable env-fix script — stays unreachable

Grepping .github/ for test-hooks and for both file names returns nothing.

Change

src/tokenless/Makefile only — no production code and no workflow file:

  • Split the pair into test-hook-scripts and hang it off test-integration, the hook-contract target the Test tokenless job already calls, exactly the way test-claude-code-detect and test-toon-cleanup ride on it.
  • test-hooks takes the new target as a prerequisite, so make still runs both suites before run-all-tests.sh and its behaviour is unchanged.
  • .PHONY for the new target is declared next to it instead of being appended to the shared list at the top of the file, which several components extend at once.
  • make help gains a row for it.

Test report

Environment: Linux x86_64, GNU bash 4.4.20, GNU Make 4.2.1, python3 3.8.17, node v22.21.1, rustc/cargo 1.96.0, jq 1.6, shellcheck 0.10.0. Base: main @ a9a3719.

The new target

  • make -C src/tokenless test-hook-scripts → exit 0, "run-hook install scope test passed" + "tool-ready hard bypass test passed".
  • make -C src/tokenless -n test-integration → prerequisite order is hook golden parity (cargo build -p tokenless-cli + test_hook_parity.py) → detect retry → toon cleanup → the two new suites → the Python contract, i.e. they run before the Python step and need nothing from it.
  • make -C src/tokenless -n test → each of the two scripts appears exactly once, so the aggregate does not double-run them through test-hooks.
  • PATH="<repo-root>/src/tokenless/target/debug:$PATH" make -C src/tokenless test-hooks → exit 0: both suites plus run-all-tests.sh 70/70, confirming the prerequisite refactor left test-hooks identical.

The "no new CI dependencies" claim, verified rather than assumed: both suites run under env -i with a PATH containing only bash and coreutils (mktemp, chmod, cp, ln, rm, mkdir, cat, dirname, grep, sed) and an empty HOME — both pass, and that same environment reports no jq, no node, no rtk and no python3. So the step the Test tokenless job now runs needs nothing a bare checkout does not already have.

Mutation checks — the suites are real guards, not decoration (each mutation reverted afterwards):

mutation result
Reorder run-hook.sh's CANDIDATES to prefer the home-directory install root test-run-hook-install-scope.sh exits 1
Delete tool_ready_hook.sh's hard bypass (printf '{}'; exit 0) test-tool-ready-readable-fixer.sh exits 1

For accuracy: dropping the -x gate in resolve_binary does not fail the second suite — what it pins is the bypass itself, not that gate.

Wider target: make -C src/tokenless test-integration (the exact CI command) → hook golden parity OK (2 tests), then the Python suites: test_compress_response_hook 39 skipped, test_codex_response_diagnostics 4 OK, test_hermes_lifecycle 13 OK, test_compress_schema_hook 18 skipped. It stops at tests/test_hook_contract.py with TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' — that file annotates with PEP 604 str | None, which needs Python >= 3.10, and the local interpreter is 3.8.17 while CI pins 3.11. Pre-existing local toolchain limit, not caused by this change (this PR touches one Makefile).

Rust sanity on the same checkout: cargo test --workspace -- --test-threads=1 → 811 passed / 0 failed / 3 ignored; cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings clean.

Merge cleanliness: git merge-tree against the heads of every open tokenless PR (#2322, #2452, #2530, #2532, #2877, #3242) — no conflicts. #2322 also edits this Makefile and its review asks for new targets to declare .PHONY separately rather than extend the shared line; this PR follows that.

Known pre-existing flake, fixed separately: while running test-integration on an unmodified main, scenario 14 of tests/test-claude-code-detect-retry.sh failed once (the 2s backoff should have been clamped to the 1s window, requested 0.000s) — a 1s window measured with $SECONDS' whole-second resolution. It is a host-speed race unrelated to this change and is fixed in #3315.

Not run: the Test tokenless CI job itself (no self-hosted runner locally).

tests/test-run-hook-install-scope.sh and
tests/test-tool-ready-readable-fixer.sh were reachable only through
`make test-hooks`, and no workflow calls that target: its third member,
tests/run-all-tests.sh, drives the release binary installed on PATH by
design (see the note above its test 5.0b), so the target as a whole
needs an installed tokenless and stays a developer-machine check.

Two contracts therefore ran nowhere but developer machines:

- run-hook.sh must resolve a hook from its own adapter tree before the
  FHS and home-directory fallbacks, so concurrent RPM, Makefile and raw
  installs cannot cross-load another version.
- tool_ready_hook.sh must keep its hard bypass, so the dormant legacy
  four-phase flow -- which runs any readable env-fix script -- stays
  unreachable.

Split the pair into `test-hook-scripts` and hang it off
`test-integration`, the hook-contract target the Test tokenless job
already calls, the same way test-claude-code-detect and test-toon-cleanup
ride on it. `test-hooks` keeps its behaviour by taking the new target as
a prerequisite, so make still runs both suites before run-all-tests.sh
and `make test` still runs each of them exactly once.

Both are plain bash over a mktemp sandbox with generated fixtures: no
cargo build output, node, jq, rtk, python3 or network, verified by
running them under a PATH holding only bash and coreutils. They cost
well under a second, so the CI step they now run in needs no new setup.
@github-actions github-actions Bot added the component:tokenless src/tokenless/ label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:tokenless src/tokenless/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant