Skip to content

Allow benchup.io in the bridge and record the live agent round - #87

Open
furilo wants to merge 1 commit into
BenchUp3D:mainfrom
furilo:live-validation-issue-55
Open

furilo wants to merge 1 commit into
BenchUp3D:mainfrom
furilo:live-validation-issue-55

Conversation

@furilo

@furilo furilo commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

This covers most of item 3 of #55, which reruns the agent scenarios from #41 against the live app. It also fixes a bug that the rerun found.

Bridge origin

https://benchup3d.github.io/benchup/ now redirects to https://benchup.io, so the published web app loads from that origin. The bridge only allowed https://benchup3d.github.io and answered the published app with HTTP 403, so once this version is deployed the web app could never have paired. The bridge now allows both origins. bridge_tests.rs covers the new origin and lookalikes of it, and docs/cli.md says so.

Live agent round

evals/setup.py gains --live --token <token> [--port <port>]. The prompts then tell each agent that the project is open in BenchUp and that it should use benchup … --live. New-project cases get an empty project to open. The requests are unchanged, and prompts without --live are the same as before.

results/round-7-live.md reran cases 01–12 that way, one at a time, on two setups. The first was a desktop build of main on macOS, with an isolated data directory for each case. The second was the web build served by trunk in Chrome 157, using its own bridge port and token.

Round 5 (files) Desktop, live Chrome, live
Commands (errors) 258 (9) 266 (4) 255 (1)
Result 10 ✅, 2 ⚠️ 11 ✅, 1 ⚠️ 11 ✅, 1 ⚠️

The ⚠️ is case 10, which is partial on purpose in every round. No agent hit a bridge error or a timeout. Each result was checked afterwards with check_model and an iso render taken from the app.

The round also checked several app behaviours, and all of them worked:

  • A batch is one undo step, Agent: ….
  • undo --live and redo --live step through the history.
  • --dry-run leaves the open model and its history untouched.
  • The web app reconnects by itself after a page reload, and keeps the change.
  • Opening another project with unsaved agent changes shows the usual unsaved-changes modal.

The tool and guide findings are listed in the results file and will be addressed in a separate PR.

Still open for item 3

  • Safari: this has not been run yet. In particular, it is still unknown whether Safari lets an https page open ws://127.0.0.1.
  • Chrome from https://benchup.io: the socket waits for the local network access permission. Nobody answered the prompt in this run, so it failed after about 240 s. The next step is to allow the prompt and confirm that the connection opens.
  • Not exercised: the grant/deny prompt, Take control, and shared projects with leases.

Testing

cargo test --locked -p benchup-cli passes.

The GitHub Pages address of the web app now redirects to
https://benchup.io, so the published app connects to the bridge from that
origin. The bridge only allowed https://benchup3d.github.io and answered it
with HTTP 403, which would have kept the published app from ever pairing.
Both origins are now allowed, with a test.

The agent evaluation can now run against the project open in BenchUp:
setup.py --live --token writes prompts that use `benchup ... --live`, and
gives new cases an empty project to open. Round 7 reran cases 01-12 this
way on the desktop and in Chrome, for item 3 of BenchUp3D#55; Safari and the local
network access prompt from an https page are still to be checked.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant