What is your issue?
Every push-to-main run of Benchmark compare last release concludes success in ~30 seconds, but asv exits before selecting an environment and no benchmark is ever executed — the compare-against-last-release performance alarm is silently dead. From today's green run (32095799355, 30 s):
· Unknown environment type 'rattler'. Allowed values based on existing plugins are ['existing', 'conda', 'uv', 'virtualenv'].
· No environments selected
Cause. #10866 (2025-10-22, 19f2973) switched asv_bench/asv.conf.json to "environment_type": "rattler" and added py-rattler to ci/requirements/environment-benchmark.yml#L15 — but benchmarks-last-release.yml builds its environment from ci/requirements/environment.yml, which never got py-rattler, so the asv it installs has no rattler plugin. The job can't see asv's non-zero exit: the run step pipes asv continuous … | sed … | tee benchmarks.log without pipefail, and the only failure gate is the grep at benchmarks-last-release.yml#L65-L70 — "Traceback \|failed\|PERFORMANCE DECREASED" — which an environment-construction failure matches never, so the step exits 0 and the run stays green.
Run-history evidence (public Actions history, verified 2026-08-18): all 450 retained runs (2025-07-02 → today) completed in ≤ ~4 minutes — none long enough to have executed the suite (a real asv continuous in the sibling Benchmark workflow takes 60–90 min, e.g. 31615795501 on #11239). 448 concluded success; the only two non-green runs (21605156467, 21606708075, 2026-02-02) were cancelled during environment setup and never reached asv. Log spot-checks across the 90-day retention window all show the identical rattler error under a green run: 26782017237 (06-01), 28122316168 (06-24), 32095799355 (08-18). Older logs have expired, so I can't state the failure mode before #10866 — but the durations show no retained run ever executed benchmarks on either side of it.
The label-triggered Benchmark workflow is not affected: its env file got py-rattler in #10866 and it genuinely executes the suite.
Verification on a fork (main synced with c2998a7): two workflow_dispatch runs of the unmodified workflow reproduce the green no-op with the same log lines: 32199849622, 32199928875.
Proposed fix — add py-rattler to ci/requirements/environment.yml, the same one-line change #10866 made to environment-benchmark.yml. Separately, set -o pipefail on the run step (or checking asv's exit status directly) would have turned this red in October instead of green for ten months. Happy to open a PR for either or both if preferred.
AI disclosure: this issue was researched and written by an AI agent, explicitly instructed to investigate and file it; all run IDs, durations, log lines, and diffs quoted above were verified against the public GitHub API before filing, and the fork runs were executed as part of that verification. [This is Claude Code on behalf of @glaziermag]
What is your issue?
Every push-to-main run of
Benchmark compare last releaseconcludes success in ~30 seconds, but asv exits before selecting an environment and no benchmark is ever executed — the compare-against-last-release performance alarm is silently dead. From today's green run (32095799355, 30 s):Cause. #10866 (2025-10-22, 19f2973) switched
asv_bench/asv.conf.jsonto"environment_type": "rattler"and addedpy-rattlertoci/requirements/environment-benchmark.yml#L15— butbenchmarks-last-release.ymlbuilds its environment fromci/requirements/environment.yml, which never gotpy-rattler, so theasvit installs has no rattler plugin. The job can't see asv's non-zero exit: the run step pipesasv continuous … | sed … | tee benchmarks.logwithoutpipefail, and the only failure gate is the grep atbenchmarks-last-release.yml#L65-L70—"Traceback \|failed\|PERFORMANCE DECREASED"— which an environment-construction failure matches never, so the step exits 0 and the run stays green.Run-history evidence (public Actions history, verified 2026-08-18): all 450 retained runs (2025-07-02 → today) completed in ≤ ~4 minutes — none long enough to have executed the suite (a real
asv continuousin the siblingBenchmarkworkflow takes 60–90 min, e.g. 31615795501 on #11239). 448 concluded success; the only two non-green runs (21605156467, 21606708075, 2026-02-02) were cancelled during environment setup and never reached asv. Log spot-checks across the 90-day retention window all show the identical rattler error under a green run: 26782017237 (06-01), 28122316168 (06-24), 32095799355 (08-18). Older logs have expired, so I can't state the failure mode before #10866 — but the durations show no retained run ever executed benchmarks on either side of it.The label-triggered
Benchmarkworkflow is not affected: its env file gotpy-rattlerin #10866 and it genuinely executes the suite.Verification on a fork (main synced with c2998a7): two
workflow_dispatchruns of the unmodified workflow reproduce the green no-op with the same log lines: 32199849622, 32199928875.Proposed fix — add
py-rattlertoci/requirements/environment.yml, the same one-line change #10866 made toenvironment-benchmark.yml. Separately,set -o pipefailon the run step (or checking asv's exit status directly) would have turned this red in October instead of green for ten months. Happy to open a PR for either or both if preferred.AI disclosure: this issue was researched and written by an AI agent, explicitly instructed to investigate and file it; all run IDs, durations, log lines, and diffs quoted above were verified against the public GitHub API before filing, and the fork runs were executed as part of that verification. [This is Claude Code on behalf of @glaziermag]