Repository navigation
test(e2e): add deterministic end-to-end suite and required CI check - #41
Merged
Merged
Conversation
Install the packed npm CLI into throw-away projects for every host and edition, drive the installed Portable workflow with a fake host Agent that follows the resume protocol using recorded stage outputs, and exercise the Classic CodeBuddy auto-dispatch and observability hooks through the local dashboard. Covers resume after crash/interruption, the requirement confirmation gate, review rejection, unfilled artifacts and repeated failures. No network access or API keys are needed. Adds .github/workflows/e2e.yml with a stable 'e2e-required' gate job and runs the suite before releases. Refs Tencent#31
cocofhu
self-requested a review
October 11, 2026 14:21
This was referenced Oct 11, 2026
This was referenced Oct 11, 2026
7 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Closes the P0 part of #31: merges currently rely on static checks, install smoke tests and manual testing. No test installs the package the way users do and then runs the workflow end to end. This PR adds a deterministic E2E suite,
bash scripts/e2e.sh, and a CI workflow so the suite can be required before a merge and a human reviews after it passes.How it stays deterministic (no API keys, no real Agent CLI)
tests/e2e/fake_agent.pystands in for the host Agent. It loads the installed Skill and follows the sameresume-driven protocol a real host follows:init,prepare --emit-prompt,assign,start,finish,approve --user-confirmed,decide-knowledgeandrotate-team. The only difference is that it writes stage outputs recorded infixtures/scenario-medium.jsoninstead of calling an LLM.--crash-in,--stop-after,--no-user-confirmation,--review-reject,--leave-placeholders,--fail-stage) drive the failure paths.npm pack, installed into a temporary prefix, which is the same path users take withnpx loopforge-cli.Scenarios (13 tests)
test_e2e_install.py)status --json,doctor, idempotentupdate, cleanuninstallthat keeps user files. Rejects Classic+pi and a second edition without--force.test_e2e_portable_workflow.py)continue-executor; codebuddy:--fresh-team→rotate-team). Resume after an interruption between actions, with no stage run twice. The requirement gate blocks DESIGN until the user confirms. A rejected review goes back to IMPLEMENT. Unfilled artifacts are rejected. Repeated failures end inblocked.test_e2e_classic_runtime.py)build_dashboard_data.py, then the dashboard served over HTTP. The test checks sessions, tool calls, tokens, the devflow run and the auto-dispatch stats, with CLS env vars removed so the run stays offline.CI
.github/workflows/e2e.yml: runs on Python 3.8 and 3.12 with Node 22 and a 15-minute timeout. Ane2e-requiredgate job gives branch protection one stable check name.release.ymlruns the suite before publishing.bash scripts/e2e.sh.Maintainer action needed: a PR can't change branch protection. Please mark
e2e-requiredas a required status check onmain, ideally together withvalidateandsecrets.Scope
Tests and CI only. Runtime files are unchanged, and
tests/e2eis not part of the npm package (checked withnpm pack --dry-run).Compatibility
No runtime behavior change. The suite uses only the Python standard library (3.8+), Node.js 20+ and npm.
Validation
Run locally on Linux, with the CLI installed from
npm pack:Mutation check: with
required_approval_gatesset to[],test_requirement_gate_waits_for_explicit_user_confirmationfails (expected exit 3, got 0).Not run: macOS runners and real Agent CLIs.
Follow-ups (separate PR)
manual-mode gates.test_cls_sink, 1 intest_collector.logs/cls-push-debug.ndjsoninto the source tree..npmignoreoverrides.gitignore, so the log can end up in a published package. The npm package also ships the observabilitytests/folders.Safety