Skip to content

refactor(benchsdk-runner): migrate from @benchsdk/client umbrella to @benchsdk/api + @benchsdk/worker - #337

Merged
dtice25 merged 15 commits into
devin/benchsdk-v2-migrationfrom
devin/benchsdk-runner-migration
Aug 21, 2026
Merged

dtice25 merged 15 commits into
devin/benchsdk-v2-migrationfrom
devin/benchsdk-runner-migration

Conversation

@dtice25

@dtice25 dtice25 commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Completes the SDK split started in #336 by moving @benchsdk/runner off the umbrella @benchsdk/client package and onto the focused @benchsdk/api and @benchsdk/worker packages.

  • runner.ts now imports createBenchmarkClient from @benchsdk/api and calls the standalone runWorker(client, options) exported by @benchsdk/worker, instead of client.runWorker(...).
  • bench-config.ts and scoring.ts import shared types (JsonObject, TaskResultRecord, etc.) from @benchsdk/api and BaseParticipant from @benchsdk/worker.
  • Test mocks were split to match the new package boundaries (@benchsdk/api for the client factory, @benchsdk/worker for runWorker/BenchmarkReporter/selectParticipants/filterParticipantsByEnv).
  • create-bench scaffolding was updated to use @benchsdk/runner ^0.2.0 so newly scaffolded projects track the post-split version.

Why

@benchsdk/runner is the orchestrator and should not reach through the backwards-compatible umbrella to the worker engine. This makes the layering explicit: @benchsdk/api is the typed REST client, @benchsdk/worker is the execution runtime, and @benchsdk/runner is the local CLI orchestrator that coordinates them.

Verification

  • pnpm -r --filter './packages/**' run build
  • pnpm typecheck
  • pnpm --filter @benchsdk/client test
  • pnpm --filter @benchsdk/runner test
  • pnpm --filter create-bench test

All passed.

Link to Devin session: https://app.devin.ai/sessions/e5c1507360b24f6f89327ccc975ae961
Requested by: @dtice25


Open in Devin Review

…@benchsdk/api + @benchsdk/worker

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25 dtice25 self-assigned this Aug 19, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 19, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

dtice25 and others added 4 commits August 19, 2026 00:26
…score

- Adds BenchmarkScoringConfig and scoringConfigToSpec in @benchsdk/runner.

- validateBenchmarkScoringConfig is wired into defineBenchmarkConfig.

- runner upserts benchmark.config.scoring and falls back to scoringConfigToSpec when onScore is omitted.

- create-bench scaffold now includes a scoring example.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
….dimensions support

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e runs

Add scoring.groupBy to split task records by a dimension key, producing one summary row per group with its own metrics and composite score. Update storage.bench.ts to run multiple file sizes in one run via phases and tag each record with file_size.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 4 commits August 19, 2026 19:02
…ht score group

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ring

Adds scoring.success.requireData so callback success predicates become serializable, and moves browser, browser-throughput, snapshot-fork, and the four ai-gateway benchmarks off onScore.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 2 commits August 19, 2026 20:45
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 2 commits August 20, 2026 13:34
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ggregate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

dtice25 and others added 2 commits August 20, 2026 15:49
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25
dtice25 merged commit 8ba174e into devin/benchsdk-v2-migration Aug 21, 2026
7 checks passed
dtice25 added a commit that referenced this pull request Aug 21, 2026
…enchsdk/api + @benchsdk/worker (#356)

* refactor(benchsdk): split SDK into @benchsdk/api, @benchsdk/worker, and @benchsdk/client umbrella

- Add @benchsdk/api: typed REST client and shared platform types.
- Add @benchsdk/worker: runWorker runtime, BenchmarkReporter, metrics, participants.
- Keep @benchsdk/client as a backwards-compatible umbrella that re-exports from the focused packages and exposes client.runWorker().
- Update pnpm workspace and lockfile for the two new packages.
- Adjust public-api.contract.test.ts to parse the new barrel .d.ts shape with workspace re-exports.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: build all workspace packages so new @benchsdk/api and @benchsdk/worker artifacts are available

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: pass composed umbrella client to runWorker so WorkerFinishContext.client matches type, and build all packages in release workflow

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(scale): copy and build the new benchsdk workspace packages in the scale image

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(benchsdk-runner): migrate from @benchsdk/client umbrella to @benchsdk/api + @benchsdk/worker (#337)

* refactor(benchsdk-runner): migrate from @benchsdk/client umbrella to @benchsdk/api + @benchsdk/worker

* feat(benchsdk-runner): declare scoring config, upload spec, and auto-score

- Adds BenchmarkScoringConfig and scoringConfigToSpec in @benchsdk/runner.

- validateBenchmarkScoringConfig is wired into defineBenchmarkConfig.

- runner upserts benchmark.config.scoring and falls back to scoringConfigToSpec when onScore is omitted.

- create-bench scaffold now includes a scoring example.

* feat(benchmarks/sandbox/tti): use new scoring config for composite score

* feat(storage): use scoring config and dimensions; add BenchmarkConfig.dimensions support

* feat(runner,storage): groupBy dimension scoring and multi-size storage runs

Add scoring.groupBy to split task records by a dimension key, producing one summary row per group with its own metrics and composite score. Update storage.bench.ts to run multiple file sizes in one run via phases and tag each record with file_size.

* fix(benchsdk-runner): keep failures and empty participants in the right score group

* refactor(benchmarks): migrate remaining benchmarks to declarative scoring

Adds scoring.success.requireData so callback success predicates become serializable, and moves browser, browser-throughput, snapshot-fork, and the four ai-gateway benchmarks off onScore.

* feat(benchsdk-runner): apply --iterations to each phase

* fix(benchsdk-runner): keep individually sized phases over --iterations

* feat(storage): run every file size in one CI run

* fix(ci): derive storage result sizes from artifact size directories

* feat(runner): include scoring spec in run summary and emit run-wide aggregate

* fix(storage): avoid duplicate summary rows for single-size runs

* feat(runner): send resolved run config to platform at run creation

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant