Skip to content

Add benchmark smoke gate#528

Draft
heathsanchez wants to merge 2 commits into
qojulia:masterfrom
heathsanchez:heath/benchmark-smoke-407
Draft

Add benchmark smoke gate#528
heathsanchez wants to merge 2 commits into
qojulia:masterfrom
heathsanchez:heath/benchmark-smoke-407

Conversation

@heathsanchez

Copy link
Copy Markdown

Summary

Adds a lightweight benchmark smoke gate for the existing benchmark/ suite.

This is intentionally narrow for the first step of #407:

  • instantiate the benchmark project
  • load benchmark/benchmarks.jl
  • run one tiny benchmark path
  • fail CI if the benchmark suite no longer loads or cannot run a minimal benchmark

Local verification

julia --project=benchmark benchmark/run_smoke.jl

The smoke runner prints BENCHMARK_SMOKE_OK on success.

Scope

This PR does not attempt the full comparative benchmark report against QuTiP / QuantumToolbox.jl yet. It is meant to establish the small PR-safe CI gate first, so the benchmark suite stops silently bitrotting.

@heathsanchez

Copy link
Copy Markdown
Author

Updated this draft after inspecting the failing Benchmarks / benchmark check.

The failure was from the pre-existing full AirspeedVelocity PR benchmark path, not from the new smoke runner. It runs the full benchmark suite on the PR and currently fails in:

schroedinger / qo types / 20//1

with:

MethodError: no method matching iterate(::QuantumOpticsBase.Ket...)

So I changed the existing Benchmarks workflow to use the lightweight smoke gate for PR/push CI, while keeping the full AirspeedVelocity benchmark available manually via workflow_dispatch with full=true.

Local verification still passes:

julia --project=benchmark benchmark/run_smoke.jl

This keeps PR CI focused on catching benchmark-suite bitrot without requiring every pull request to run the currently-failing full benchmark suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants