fix: make Q1-Q8 matrix and HF aggregation fail closed - #28
Merged
Conversation
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change closes the software-side gaps in the Q1–Q8 single-node matrix:
matrix_status.json;q1–q8at the schema and aggregation boundaries;mainbranch and use the official write endpoint;The benchmark workflow also exposes backend, seed, and parallelism inputs, keeps
the single-node contract explicit, aligns the embedding smoke-test model with
the launched model, and reports an early sagellm exit without waiting six
minutes.
Why
The Q1–Q8 runner and schema were already present on
main, but a failedworkload still returned success and Q IDs were written with inconsistent case.
That allowed CI and HF aggregation to present an incomplete matrix as a
successful one. The HF workflow also targeted the removed
main-devbranch, soit could not be triggered by the repository's current branch layout.
Recent nightly runs still fail before the benchmark because the configured HF
mirror cannot serve the tiny model to the sagellm preflight. This PR switches
that read path to the official endpoint, but deliberately does not claim the
CI acceptance criterion until a post-merge or branch-equivalent workflow run is
green.
Validation
python3 -m pytest -q tests/test_metrics_schema.py tests/test_q_matrix_runner.py tests/test_hf_aggregation.py tests/test_legacy_terms.py --override-ini='addopts='— 10 passedruff check .andruff format --check .passedgit diff --checkpassedThe first PR run exposed 22 pre-existing Ruff failures that also fail on current
main: 11 import-order findings, 10 staleFlownetEnvironmentannotationreferences, and one missing
StopSignalimport. The follow-up commit appliesmechanical import ordering, renames only those annotations to the already
imported
FluttyEnvironment, and restores the missing runtime import. It doesnot change benchmark policy or execution logic.
Closes #11
Closes #12
Closes #15
Refs #10
Refs #13
Refs #14