Skip to content

fix: make Q1-Q8 matrix and HF aggregation fail closed - #28

Merged
ShuhaoZhangTony merged 3 commits into
mainfrom
codex/resolve-q-matrix-issues
Jul 30, 2026
Merged

fix: make Q1-Q8 matrix and HF aggregation fail closed#28
ShuhaoZhangTony merged 3 commits into
mainfrom
codex/resolve-q-matrix-issues

Conversation

@ShuhaoZhangTony

@ShuhaoZhangTony ShuhaoZhangTony commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

This change closes the software-side gaps in the Q1–Q8 single-node matrix:

  • return a non-zero exit code when any workload repetition fails;
  • support explicit continue-on-error and fail-fast policies;
  • write per-run logs and an incremental matrix_status.json;
  • canonicalize Q-matrix workload IDs to q1q8 at the schema and aggregation boundaries;
  • reject malformed local result records instead of silently skipping them;
  • publish deterministic coverage counts by workload, backend, and seed;
  • restore the HF upload trigger to the existing main branch and use the official write endpoint;
  • document the reproducible single-node, CI, aggregation, and troubleshooting flows.

The benchmark workflow also exposes backend, seed, and parallelism inputs, keeps
the single-node contract explicit, aligns the embedding smoke-test model with
the launched model, and reports an early sagellm exit without waiting six
minutes.

Why

The Q1–Q8 runner and schema were already present on main, but a failed
workload still returned success and Q IDs were written with inconsistent case.
That allowed CI and HF aggregation to present an incomplete matrix as a
successful one. The HF workflow also targeted the removed main-dev branch, so
it could not be triggered by the repository's current branch layout.

Recent nightly runs still fail before the benchmark because the configured HF
mirror cannot serve the tiny model to the sagellm preflight. This PR switches
that read path to the official endpoint, but deliberately does not claim the
CI acceptance criterion until a post-merge or branch-equivalent workflow run is
green.

Validation

  • python3 -m pytest -q tests/test_metrics_schema.py tests/test_q_matrix_runner.py tests/test_hf_aggregation.py tests/test_legacy_terms.py --override-ini='addopts=' — 10 passed
  • repository-wide ruff check . and ruff format --check . passed
  • Python byte-compilation passed for changed runtime scripts
  • Both modified workflow files parsed successfully as YAML
  • git diff --check passed

The first PR run exposed 22 pre-existing Ruff failures that also fail on current
main: 11 import-order findings, 10 stale FlownetEnvironment annotation
references, and one missing StopSignal import. The follow-up commit applies
mechanical import ordering, renames only those annotations to the already
imported FluttyEnvironment, and restores the missing runtime import. It does
not change benchmark policy or execution logic.

Closes #11
Closes #12
Closes #15
Refs #10
Refs #13
Refs #14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant