Repository navigation
feat(e2e): add per-family end-to-end regression runner - #9894
Pfannkuchensack wants to merge 22 commits into
Conversation
Runs each installed model family through a launched scratch server and compares PSNR, time and driver-level VRAM against accepted references. Graphs come from the frontend builders via an offline graph export; the generate submission plan is shared with the workbench.
github-actions
Bot
added
frontend-deps
Main added useNsfwChecker and useWatermark to the generation project settings; exported calibration and E2E graphs run without both, as the defaults do.
Adds `bundle`, which packs a run's report, images, graphs and traces into a zip with local paths, credentials and PNG text metadata removed. Runs record name, base, type, format, variant and hash of every model a case uses; report and summary list them per case. AGENTS.md asks agents to offer a family run after model-related changes.
Each bundle carries bundle.json with format version, tool commit and creation time. The previous-run search filters by profile from the run id before loading, so runs of other profiles are not read.
The fake sampler now rises when the run is enqueued instead of halfway to stop, so a slow runner cannot move the peak past completion.
Cases can declare `expect: changed`; a difference from the reference then reports CHANGED, and an unchanged output fails. A run judged against its own accepted reference reports BASELINE; bundles move to format version 2. The server-tree test retries reading its pid file instead of failing on a briefly locked file.
Bundles carry reference/<case_key>/reference.json next to the reference image, with its environment sanitized like run.json. URLs with local-network hosts are replaced by <local host> in all bundle text.
Add an NVML backend with the WDDM sampler's interface; on Linux the runner uses it and pins the server to the selected GPU. Linux adapters are keyed by UUID for locks and selection, so identical cards can run side by side; records never store the UUID. Bundles also redact POSIX paths in free text.
The cache logs a tokenizer at 0.0% of 0.03MB, which made every FLUX record invalid. Residency records the logged total size, and loads under 1 MB count as resident in the checks and the report.
| else: | ||
| roots.append(Root("<template_root>", str(manifest.server.template_root), True)) | ||
| roots.append(Root("<home>", str(Path.home()), False)) | ||
| env = {name: value for environment in environments for name, value in environment.extra_env.items()} |
There was a problem hiding this comment.
P1 Preserve every credential value when combining environments
When the current run and an accepted reference contain the same environment-variable name with different values, this dictionary keeps only the reference’s value because reference environments come last. For example, rotating HF_TOKEN between the baseline and current run removes the current token from the sanitizer’s replacement list. Any occurrence in an error or other exported text then survives into the shareable bundle. Executing the pinned sanitizer with two different HF_TOKEN values reproduced this: the old token was redacted while the current token remained verbatim. Collect all distinct credential name/value pairs instead of collapsing them by variable name.
There was a problem hiding this comment.
Fixed in a43c2ba. The sanitizer now collects every distinct name/value pair of credential-named variables from the run, every accepted reference and the previous runs the report compares against, instead of collapsing them by name, and redacts each value (placeholder <NAME>, deterministic when one value appears under several names).
test_a_token_rotated_across_the_compared_runs_is_redacted_in_every_value gives the bundled run, its reference and its previous run three different HF_TOKEN values, echoes them in a compared setting and in a record error, and asserts that none of them appears in any zip entry. The test fails against the previous code.
joshistoast
left a comment
joshistoast
left a comment
There was a problem hiding this comment.
Some environment and cleanup defects
| tree = [] | ||
| if job is not None: | ||
| job.close() | ||
| elif process.poll() is None: |
There was a problem hiding this comment.
[P2] Clean up Linux descendants after their launcher exits
Cleanup skips the process-group kill after the launcher exits. Surviving children can become reparented and evade descendant discovery, retaining GPU memory or files and obstructing recovery. The reviewer reproduced a child remaining alive after its parent exited. Cleanup should retain ownership of the launched group and handle surviving members.
There was a problem hiding this comment.
Fixed in a43c2ba. On POSIX, terminate() now always kills the launched process group (killpg, ignoring ESRCH) instead of skipping it once the leader has exited. It then waits for the captured tree and only reaps the leader afterwards.
Liveness checks (alive(), the startup check, the crash path) use waitid(..., WNOWAIT) instead of poll(). An exited leader therefore stays a zombie and keeps its group id reserved until the kill, so the group cannot be confused with an unrelated one.
stop also kills the groups led by the processes it identified by pid and create time. That covers descendants that were reparented out of the tree.
New POSIX-only tests cover a leader that exited (both reaped and still unreaped), a grandchild orphaned before stop, and a dead leader that stays unreaped until terminate(). They run in the Linux and macOS CI jobs.
…esses Bundles redact each distinct credential value from the run, its references and compared previous runs, so a rotated token cannot survive. On POSIX the server's process group is always killed, and liveness checks keep an exited leader unreaped until then; stop also kills groups led by identified processes.
Where CPython lacks os.waitid, psutil decides whether the server exited without reaping it; crash messages then report the exit code as unknown.
macOS refuses killpg with EPERM once the only member of the group is the unreaped leader; the kill now re-raises only while a member still lives.
Before the runner replaces the heartbeat or touches a scratch root, it stops the recorded server and its group, or refuses while it survives. An unreaped process counts as gone, while live members of its group are still killed.
Summary
End-to-end checks per model family are run by hand for almost every PR today. Several recent defects would have shown up in such a run early: a tokenizer that silently loaded a one-token vocabulary, missing working-memory reservations, and samplers that ignored the img2img start point.
This adds a developer tool,
python -m scripts.family_e2e, that runs each locally installed family end to end through the real API and compares the result with an explicitly accepted reference and with the previous run:scripts/family_e2e/manifest.example.yaml.pnpm graphs:export, builds the enqueue batches with the same graph builders and component policy as the Generate and Canvas widgets. The generate submission planning (prompt templates, negative-prompt suppression, dynamic prompts, seed plan, batch data) moved into Generation and Queue and is now shared by the workbench and the export, so the two cannot drift.use_cache=falseon every node, the model cache is emptied before each family, then 1 warm-up and 3 measured runs per case and size.accept.status,stop,resume, a per-GPU lock, and a kill-on-close job object so the server never outlives its runner.Records keep per-node baseline and peak plus the raw memory trace, so several sizes per family can later feed working-memory calibration without a second run.
bundlepacks a run into a zip that opens on any machine. It holds the report and summary, rendered for the zip, with every shown image (this run, reference, previous run, difference) linked relatively, plusrun.json,records.jsonl,graphs/andtraces/. Abundle.jsonholds the zip's format version, tool commit and creation time, so readers can refuse layouts they don't know.run.jsonrecords name, base, type, format, variant, hash and a shareable source for every model a case's graph uses. The report and summary list them per case, since the local model key means nothing on another machine. Older runs without this field still load, report and bundle.AGENTS.mdasks agents to offer a family run of the affected families after changes to model loading, quantization, attention, the model cache, model-running invocations or the frontend graph builders. They should never start one unasked, and the result goes into the PR as the summary table plus the bundle zip.QA Instructions
Checks
uv run --no-sync pytest tests/family_e2e -n 4: 215 passed on Windows. Theslowreal-GPU sampler tests passed on an RTX 4090 (WDDM) and on both GPUs of a 2× RTX PRO 6000 Linux machine (NVML).tests/family_e2e -m ""passes with only the Windows-only tests skipped.src/hrefresolves inside it;uv tool run ruff@0.11.2 check/format --checkonscripts/family_e2e tests/family_e2e: clean.invokeai/frontend/webv2):pnpm lintpasses, includingarchitecture:check.pnpm test:fixtures: 35 passed.pnpm test:performance:buildpasses with no baseline or pin changes.pnpm test: 10,276 passed. Known failures on this machine, not caused by this change:clusterStats,indexProgress,modelCacheStore.dependencyPolicy,typographyPolicy,workbenchOwnership) intermittently hit their 30 s / 60 s timeouts. They pass alone and insidepnpm lint.Graph.model_validate+validate_self()andBatch.model_validate: txt2img for every coverage shape of all 14 generate bases, img2img for 12 bases, ControlNet for 5. ERNIE-Image and Ideogram 4 correctly report that canvas generation is not supported.Real hardware
starton the same GPU is refused by the lock.stopandresumerepeat the warm-up and add only the missing measurements.batch.dataat the expected node paths.Try it
manifest.example.yaml, fill in your model keys, and stop the dev server for the template root.python -m scripts.family_e2e start --manifest <file>, thenstatus,accept <run>,report <run>, andbundle <run>for a shareable zip.First findings from the tool
Review
Resolved
device: cuda:N/generation_deviceslist to match.Remaining limits
/api/…and/ws/…paths stay so tool errors keep their endpoint.image_sha256in a bundled record refers to the original PNG, before its metadata was removed.Checklist
What's Newcopy (if doing a release after this PR)