Skip to content

test(e2e): cover all routes/modes, render dashboard in CI, fix issues found (P1 of #31) - #42

Merged
cocofhu merged 1 commit into
Tencent:mainfrom
Yanami-2K:test/e2e-p1
Oct 11, 2026
Merged

cocofhu merged 1 commit into
Tencent:mainfrom
Yanami-2K:test/e2e-p1

Conversation

@Yanami-2K

@Yanami-2K Yanami-2K commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Refs #31. Follow-up to #41 (merged); rebased onto main, so the diff shows only this PR's commit.

What this adds (P1 of the #31 plan)

More E2E scenarios. All of them use the fake host Agent and need no network or API key.

  • test_e2e_workflow_modes.py:
    • small/SOLO on Claude and CodeBuddy
    • SOLO overflow escalating to medium
    • large with a knowledge stage that actually runs (Codex)
    • manual mode, with every gate checked: REQUIREMENT needs user confirmation; DESIGN, REVIEW and TEST each need an approval; a gate nobody answers holds the next stage until a resume
    • design-implementation tasks (design context plus visual evidence required, and rejected when the evidence is missing)
  • test_e2e_dashboard_browser.py renders the observability dashboard in Chromium with Playwright:
    • it checks the data badge, KPIs, session table, Devflow run card and auto-dispatch health
    • it covers the empty state and the "snapshot failed to load" state
    • every request outside 127.0.0.1 is blocked, so the page must work offline
    • CI uploads the screenshots as the dashboard-screenshots artifact
  • Install checks: every Markdown link in an installed bundle resolves, and installs contain no runtime logs.

CI

  • The e2e matrix now runs on ubuntu with Python 3.8 + Node 22, Python 3.12 + Node 22 and Python 3.12 + Node 20, plus macOS with Python 3.12 + Node 22.
  • A new e2e-browser job also feeds the single e2e-required gate.
  • validate and release now run scripts/test-observability.sh: the 129 agent-observability unit tests plus 3 new ones, in all three Classic bundles. Afterwards the script fails if the tests left any log file in the source tree.

Fixes found by the new tests

  1. Portable SOLO overflow was a dead end. finish --result overflow left current_stage=SOLO, which is outside the new medium route, so resume, prepare and validate all rejected the escalated state. The stage now parks on the PHASE-0 sentinel. A regression unit test is in test_regressions.py.
  2. Dashboard showed unfinished runs as "已完成". A run sitting between stages (just started, or waiting for approval) has no in_progress stage, so it fell through to completed. It is now "进行中" unless every stage is completed or skipped. Unit tests added.
  3. The 6 failing observability unit tests.
    • 5 CLS sink tests depended on a Node SDK checked out next to the repo. They now run hermetically, using a stub SDK path and a temp debug log.
    • The standby-assignment test expected behaviour that was never implemented. A "待命" (standby) assignment from main now only creates the member and does not switch the active agent.
  4. Logs leaking into the source tree and into packages.
    • Running those tests wrote cls-push-debug.ndjson into .codebuddy/skills/agent-observability/logs/. After that, build-classic-hosts.py --check failed locally, and the file was copied into installs (and could ship from a dirty checkout).
    • CLS mirroring now writes nothing when CLS isn't configured.
    • Runtime log files are excluded at every layer: the installer snapshot, the classic generator, the npm files list, the wheel data_files and the release archive. .gitkeep is the only exception.
    • smoke-npm.sh asserts that the package contains no runtime log files.

Validation

  • Fork CI is all green, including macOS and Node 20: [fork CI] test(e2e) P1 — mirror of upstream P1 PR Yanami-2K/LoopForge#2
  • Locally, scripts/e2e.sh passes 23 tests, browser tests included, on Python 3.13, and passes on Python 3.8 with the browser module skipped.
  • validate.sh, test-observability.sh (132 tests, py3.8 and py3.13), smoke-install.sh, smoke-npm.sh and scan-secrets.sh all pass.

Follow-ups (not in this PR)

The generated Claude/Cursor Classic docs still mention some backtick paths that don't exist after install:

  • .claude/agents/architect.md (the file is devflow-architect.md)
  • .claude/rules/global.mdc (Claude uses .md)
  • .claude/skills/superpowers/brainstorming/SKILL.md
  • .claude/skills/devflow-codex/SKILL.md, which comes from the Codex workflows/devflow.md

These paths are in backticks, not links, so the new link check doesn't catch them. Fixing them belongs in build-classic-hosts.py's adapt(). I can open a separate issue or PR.

E2E (P1 of Tencent#31):
- small/SOLO, SOLO overflow, large with knowledge, manual gates and
  design-implementation tasks driven by the fake host Agent
- Playwright render of the observability dashboard (data, empty and
  load-failure states, offline), screenshots uploaded from CI
- installs ship no runtime logs and every Markdown link resolves
- matrix adds Node 20 and macOS; a browser job joins e2e-required

Fixes found by the new tests:
- portable: SOLO overflow left current_stage outside the medium route,
  so resume/prepare/validate rejected the escalated state
- dashboard: runs waiting between stages were shown as completed
- observability: standby team assignments no longer switch the active
  agent; CLS mirroring stays silent when CLS is not configured
- observability tests are hermetic (no source-tree logs, no Node SDK)
  and run in CI via scripts/test-observability.sh (132 tests)
- runtime logs are excluded from installs, npm package, wheel and the
  release archive

Refs Tencent#31
@Yanami-2K Yanami-2K changed the title test(e2e): cover all routes/modes, render dashboard in CI, fix issues found (P1 of #31, depends on #41) test(e2e): cover all routes/modes, render dashboard in CI, fix issues found (P1 of #31) Oct 11, 2026
@cocofhu
cocofhu merged commit 96e38f9 into Tencent:main Oct 11, 2026
11 checks passed
@Yanami-2K Yanami-2K mentioned this pull request Oct 11, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants