Skip to content

feat(benchmarks): enable Criterion automatically when a benchmark namespace is set - #26210

Open
Omega359 wants to merge 1 commit into
apache:mainfrom
Omega359:sql_benchmark_criterion_namespace
Open

Omega359 wants to merge 1 commit into
apache:mainfrom
Omega359:sql_benchmark_criterion_namespace

Conversation

@Omega359

@Omega359 Omega359 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Is your feature request related to a problem or challenge?

Running a SQL benchmark with --criterion-namespace or BENCH_NAMESPACE currently requires a separate --criterion flag. Since namespaces apply to Criterion benchmark groups, selecting a namespace should also select Criterion mode.

Describe the solution you'd like

Enable Criterion when either namespace option is set. Apply the existing mode checks after inferring Criterion mode, so incompatible --output and --iterations options still fail and --save-baseline works. Print the inferred-mode notice to stderr to preserve stdout output.

Describe alternatives you've considered

Keeping the explicit --criterion requirement would preserve the current behavior but require callers to specify both options.

Additional context

This follows the SQL benchmark migration preparation in #24806.

Willingness to contribute

Implementation included in this draft PR.

Which issue does this PR close?

Rationale for this change

Allow benchmark callers to select namespaced Criterion runs with a single option or environment variable.

What changes are included in this PR?

  • Infer Criterion mode from --criterion-namespace and BENCH_NAMESPACE and update the CLI help.
  • Share namespace validation between the CLI and the Criterion runner.
  • Extend CLI tests for inferred mode, environment defaults, CLI overrides, list and dry-run behavior, conflicting options, saved baselines, and stderr notices.

What is the testing strategy for this PR?

The benchmark-runner tests exercise argument parsing and environment handling, including a subprocess test for BENCH_NAMESPACE. Existing tests cover invalid namespaces and distinct Criterion artifacts.

Are there any user-facing changes?

Yes. --criterion-namespace and BENCH_NAMESPACE now enable Criterion without an explicit --criterion flag. The runner reports inferred mode on stderr.

Infer Criterion mode from --criterion-namespace or BENCH_NAMESPACE and report automatic activation on stderr. Preserve listing, dry-run behaviour, and Criterion option compatibility checks.

Share namespace validation between the CLI and SQL benchmark runner. Add regression coverage for mode inference, CLI precedence, and stderr output.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.34146% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.80%. Comparing base (454ca08) to head (c89713c).

Files with missing lines Patch % Lines
benchmarks/src/bin/benchmark_runner.rs 96.29% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26210      +/-   ##
==========================================
- Coverage   82.80%   82.80%   -0.01%     
==========================================
  Files        1149     1149              
  Lines      452213   452272      +59     
  Branches   452213   452272      +59     
==========================================
+ Hits       374473   374520      +47     
- Misses      54945    54953       +8     
- Partials    22795    22799       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Omega359
Omega359 marked this pull request as ready for review October 11, 2026 15:31
@Omega359 Omega359 changed the title fix: enable Criterion automatically when a benchmark namespace is set feat: enable Criterion automatically when a benchmark namespace is set Oct 11, 2026
@Omega359 Omega359 changed the title feat: enable Criterion automatically when a benchmark namespace is set feat(benchmarks): enable Criterion automatically when a benchmark namespace is set Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants