test(tokenless): run the hook script suites from test-integration - #3316
Open
Forrest-ly wants to merge 1 commit into
Open
Forrest-ly wants to merge 1 commit into
Forrest-ly wants to merge 1 commit into
Conversation
tests/test-run-hook-install-scope.sh and tests/test-tool-ready-readable-fixer.sh were reachable only through `make test-hooks`, and no workflow calls that target: its third member, tests/run-all-tests.sh, drives the release binary installed on PATH by design (see the note above its test 5.0b), so the target as a whole needs an installed tokenless and stays a developer-machine check. Two contracts therefore ran nowhere but developer machines: - run-hook.sh must resolve a hook from its own adapter tree before the FHS and home-directory fallbacks, so concurrent RPM, Makefile and raw installs cannot cross-load another version. - tool_ready_hook.sh must keep its hard bypass, so the dormant legacy four-phase flow -- which runs any readable env-fix script -- stays unreachable. Split the pair into `test-hook-scripts` and hang it off `test-integration`, the hook-contract target the Test tokenless job already calls, the same way test-claude-code-detect and test-toon-cleanup ride on it. `test-hooks` keeps its behaviour by taking the new target as a prerequisite, so make still runs both suites before run-all-tests.sh and `make test` still runs each of them exactly once. Both are plain bash over a mktemp sandbox with generated fixtures: no cargo build output, node, jq, rtk, python3 or network, verified by running them under a PATH holding only bash and coreutils. They cost well under a second, so the CI step they now run in needs no new setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
make test-hookshas three members and no workflow calls it — its third member,tests/run-all-tests.sh, drives the release binary installed onPATHon purpose (see the note above its test 5.0b), so the target as a whole needs an installedtokenlessand stays a developer-machine check. The other two members therefore ran nowhere but developer machines, and two contracts went unenforced in CI:tests/test-run-hook-install-scope.shrun-hook.shresolves a hook from its own adapter tree before the FHS and home-directory fallbacks, so concurrent RPM, Makefile and raw installs cannot cross-load another versiontests/test-tool-ready-readable-fixer.shtool_ready_hook.shkeeps its hard bypass, so the dormant legacy four-phase flow — which runs any readable env-fix script — stays unreachableGrepping
.github/fortest-hooksand for both file names returns nothing.Change
src/tokenless/Makefileonly — no production code and no workflow file:test-hook-scriptsand hang it offtest-integration, the hook-contract target theTest tokenlessjob already calls, exactly the waytest-claude-code-detectandtest-toon-cleanupride on it.test-hookstakes the new target as a prerequisite, so make still runs both suites beforerun-all-tests.shand its behaviour is unchanged..PHONYfor the new target is declared next to it instead of being appended to the shared list at the top of the file, which several components extend at once.make helpgains a row for it.Test report
Environment: Linux x86_64, GNU bash 4.4.20, GNU Make 4.2.1, python3 3.8.17, node v22.21.1, rustc/cargo 1.96.0, jq 1.6, shellcheck 0.10.0. Base:
main@a9a3719.The new target
make -C src/tokenless test-hook-scripts→ exit 0, "run-hook install scope test passed" + "tool-ready hard bypass test passed".make -C src/tokenless -n test-integration→ prerequisite order is hook golden parity (cargo build -p tokenless-cli+test_hook_parity.py) → detect retry → toon cleanup → the two new suites → the Python contract, i.e. they run before the Python step and need nothing from it.make -C src/tokenless -n test→ each of the two scripts appears exactly once, so the aggregate does not double-run them throughtest-hooks.PATH="<repo-root>/src/tokenless/target/debug:$PATH" make -C src/tokenless test-hooks→ exit 0: both suites plusrun-all-tests.sh70/70, confirming the prerequisite refactor lefttest-hooksidentical.The "no new CI dependencies" claim, verified rather than assumed: both suites run under
env -iwith aPATHcontaining only bash and coreutils (mktemp,chmod,cp,ln,rm,mkdir,cat,dirname,grep,sed) and an emptyHOME— both pass, and that same environment reports nojq, nonode, nortkand nopython3. So the step theTest tokenlessjob now runs needs nothing a bare checkout does not already have.Mutation checks — the suites are real guards, not decoration (each mutation reverted afterwards):
run-hook.sh'sCANDIDATESto prefer the home-directory install roottest-run-hook-install-scope.shexits 1tool_ready_hook.sh's hard bypass (printf '{}'; exit 0)test-tool-ready-readable-fixer.shexits 1For accuracy: dropping the
-xgate inresolve_binarydoes not fail the second suite — what it pins is the bypass itself, not that gate.Wider target:
make -C src/tokenless test-integration(the exact CI command) → hook golden parity OK (2 tests), then the Python suites:test_compress_response_hook39 skipped,test_codex_response_diagnostics4 OK,test_hermes_lifecycle13 OK,test_compress_schema_hook18 skipped. It stops attests/test_hook_contract.pywithTypeError: unsupported operand type(s) for |: 'type' and 'NoneType'— that file annotates with PEP 604str | None, which needs Python >= 3.10, and the local interpreter is 3.8.17 while CI pins 3.11. Pre-existing local toolchain limit, not caused by this change (this PR touches one Makefile).Rust sanity on the same checkout:
cargo test --workspace -- --test-threads=1→ 811 passed / 0 failed / 3 ignored;cargo fmt --all -- --checkandcargo clippy --workspace --all-targets -- -D warningsclean.Merge cleanliness:
git merge-treeagainst the heads of every open tokenless PR (#2322, #2452, #2530, #2532, #2877, #3242) — no conflicts. #2322 also edits this Makefile and its review asks for new targets to declare.PHONYseparately rather than extend the shared line; this PR follows that.Known pre-existing flake, fixed separately: while running
test-integrationon an unmodifiedmain, scenario 14 oftests/test-claude-code-detect-retry.shfailed once (the 2s backoff should have been clamped to the 1s window, requested 0.000s) — a 1s window measured with$SECONDS' whole-second resolution. It is a host-speed race unrelated to this change and is fixed in #3315.Not run: the
Test tokenlessCI job itself (no self-hosted runner locally).