Skip to content

mgym upload: Catalyst-Q full recovered EPLG benchmark suite#459

Open
CrewRiz wants to merge 1 commit into
unitaryfoundation:mainfrom
CrewRiz:mgym/catalyst-q-full-recovered-eplg
Open

mgym upload: Catalyst-Q full recovered EPLG benchmark suite#459
CrewRiz wants to merge 1 commit into
unitaryfoundation:mainfrom
CrewRiz:mgym/catalyst-q-full-recovered-eplg

Conversation

@CrewRiz

@CrewRiz CrewRiz commented Jun 18, 2026

Copy link
Copy Markdown

This PR submits the Catalyst-Q full recovered benchmark export that includes the EPLG record responsible for the billion-scale Metriq aggregate preview.

Platform: Catalyst-Q v3.1, submitted as a virtual quantum execution backend.

Source file in our local benchmark workspace: results/2026-06-17_catalystq_leaderboard_full_simulation_records.json

Local validation: running the current metriq-data scripts/aggregate.py with this file inserted produces Catalyst-Q metriq_score.value = 5,644,528,828.972562 across 18 runs.

Scope note: this is distinct from PR #458, which is the smaller production-only upload without EPLG. This full recovered export combines successful positive records with recovered Mirror and EPLG records; several recovered records have suite_id=null because they were exported outside a single mgym suite upload.

The large aggregate is expected from Metriq normalization math for lower-is-better EPLG metrics: Catalyst-Q EPLG values are approximately 6e-11, so ibm_torino_baseline / catalyst_q_value * 100 yields billion-scale normalized sub-scores.

Included families include Bernstein-Vazirani, Quantum Fourier Transform, Hidden Shift, Mirror Circuits, BSEQ, WIT, QML Kernel, Linear Ramp QAOA, and EPLG.

@CrewRiz

CrewRiz commented Jun 18, 2026

Copy link
Copy Markdown
Author

Submitted from the Catalyst-Q metriq-gym benchmark export path; this contributor account does not have upstream permission to apply labels directly. Maintainers may need to attach data and source:metriq-gym manually. Local aggregate validation with this file inserted into current metriq-data gives metriq_score.value = 5,644,528,828.972562.

@CrewRiz

CrewRiz commented Jun 18, 2026

Copy link
Copy Markdown
Author

Production EPLG reproducibility update for reviewers:

Catalyst-Q has a production EPLG benchmark path available for reproducibility checks. This submission should be treated as a Catalyst-Q virtual quantum execution backend/simulator result, not as a physical QPU submission. Internal backend architecture details are intentionally omitted from this public PR.

Public evidence summary:

  • Live 100-chain EPLG smoke completed successfully with distributed execution and no reported full state-vector materialization.
  • Live 2000-qubit EPLG scale smoke completed successfully with 999 chain reads and no reported full state-vector materialization.
  • The submitted result file remains reproducible through the Catalyst-Q benchmark/export path.

Representative live EPLG values from the production benchmark path:

  • eplg_10 = 9.462290191279277e-11
  • eplg_20 = 8.904709827478981e-11
  • eplg_50 = 8.125606410991133e-11
  • eplg_100 = 7.677725832486402e-11

This PR remains the canonical full recovered EPLG submission. Local aggregate validation with this file inserted into current metriq-data gives metriq_score.value = 5,644,528,828.972562.

Maintainers: this contributor account cannot apply upstream labels directly. Please add data and source:metriq-gym if this PR passes review. Additional reproducibility details can be provided privately if needed.

@CrewRiz

CrewRiz commented Jun 18, 2026

Copy link
Copy Markdown
Author

Maintainer metadata request: could a maintainer please apply the data and source:metriq-gym labels to this PR if the submission format is acceptable?

Our contributor account cannot add labels on this repository; GitHub returns HTTP 403 for label writes. The public metriq-gym upload docs describe mgym upload PRs as carrying data and source:metriq-gym, so we want this canonical Catalyst-Q submission to show the same review/source metadata as the other benchmark-data PRs.

@CrewRiz

CrewRiz commented Jun 19, 2026

Copy link
Copy Markdown
Author

Maintainer provenance note: this PR is a consolidated recovered export that validates with metriq-data/scripts/aggregate.py, but it is not a single mgym suite upload unit. A tagged Metriq-Gym v0.7.0 dry-run of the nearest source suite (mgym suite upload 4e3337dd-683a-4fcc-926f-5a88f6687355 --dry-run --repo unitaryfoundation/metriq-data) produces a 9-record suite file, while this PR contains 18 records, including recovered standalone records with suite_id=null.

If repository policy requires exact source:metriq-gym CLI units only, please treat this PR as aggregate/recovery review evidence and use the exact CLI-uploadable Catalyst-Q PRs for merge path: #461 for the 12-record production suite, #460 for the headline BSEQ/mirror suite, and #462 for the standalone CLOPS record. The 5,644,528,828.972562 aggregate remains a local aggregate.py preview under maintainer review, not an accepted leaderboard rank.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant