Repository navigation
feat(benchmarks): pnpm workspace + port SDK + switch consumers - #207
Conversation
Contributor License AgreementAll contributors are covered by a CLA. |
There was a problem hiding this comment.
Pull request overview
This PR converts the repo into a pnpm workspace with separate packages/bench (the benchmark orchestrator SDK) and packages/cli (the benchmark runner/consumer), then switches scale/storage/etc. CLI consumers from @computesdk/bench to the new local benchmarks workspace package and updates CI workflows accordingly.
Changes:
- Introduces pnpm workspace layout (
packages/bench,packages/cli) and migrates CI workflows to pnpm. - Ports the benchmark orchestrator client/worker helpers into
packages/benchand updates CLI code to import frombenchmarks. - Refactors benchmark runners (sandbox/storage/snapshot-fork/browser/scale) and adjusts result paths to the repo root
results/directory.
Reviewed changes
Copilot reviewed 44 out of 85 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/scale/Dockerfile | Removes old root-level scale Dockerfile (moved under packages/cli). |
| pnpm-workspace.yaml | Declares pnpm workspace packages (packages/bench, packages/cli). |
| packages/cli/tsconfig.json | Adds CLI TypeScript configuration for the new workspace package. |
| packages/cli/src/util/timeout.ts | Adds a shared promise timeout helper. |
| packages/cli/src/util/stats.ts | Adds shared percentile/stats helpers (used by sandbox benchmarks). |
| packages/cli/src/util/error.ts | Adds error formatting that preserves provider .cause chains. |
| packages/cli/src/update-readme.ts | Updates repo-root path resolution for new workspace layout. |
| packages/cli/src/storage/types.ts | Adds storage benchmark types and snapshot/fork override config shape. |
| packages/cli/src/storage/stats.ts | Adds shared rounding/stats helpers for storage benchmarks. |
| packages/cli/src/storage/snapshot-fork-types.ts | Introduces snapshot/fork dataset presets + result types. |
| packages/cli/src/storage/snapshot-fork-benchmark.ts | Implements snapshot/fork benchmark runner + scoring + JSON writer. |
| packages/cli/src/storage/scoring.ts | Adds composite scoring/sorting for storage benchmark results. |
| packages/cli/src/storage/providers.ts | Defines storage provider configs using StorageSDK adapters. |
| packages/cli/src/storage/generate-svg.ts | Updates repo-root path resolution for rendered storage SVGs. |
| packages/cli/src/storage/generate-snapshot-fork-svg.ts | Updates repo-root path resolution for rendered snapshot/fork SVGs. |
| packages/cli/src/storage/benchmark.ts | Implements upload/download/throughput storage benchmark + JSON writer. |
| packages/cli/src/scale/types.ts | Adds types for scale/burst-100k result shapes and metrics. |
| packages/cli/src/scale/sinks/tigris.ts | Adds Tigris S3-compatible sink for uploading raw/meta/metrics/logs. |
| packages/cli/src/scale/sdk-coordinator.ts | Switches orchestrator client imports to benchmarks workspace package. |
| packages/cli/src/scale/scripts/watch.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/scripts/start.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/scripts/spinup-idle.ts | Adds script to provision idle Namespace VMs for capacity dry-runs. |
| packages/cli/src/scale/scripts/kill.ts | Adds script to list/destroy live Namespace VMs for a run prefix. |
| packages/cli/src/scale/scripts/aggregate.ts | Switches orchestrator client imports to benchmarks. |
| packages/cli/src/scale/runner.ts | Adds two-phase burst lifecycle runner + structured result emission. |
| packages/cli/src/scale/providers.ts | Defines burst-100k provider participation and keep-alive settings. |
| packages/cli/src/scale/logger.ts | Adds structured logger with in-memory transcript for uploads. |
| packages/cli/src/scale/Dockerfile.norun | Adds a “bundle but don’t auto-run coordinator” scale image Dockerfile. |
| packages/cli/src/scale/Dockerfile | Adds pnpm-workspace-aware scale image Dockerfile that bundles coordinator. |
| packages/cli/src/scale/bench-reporter.ts | Switches orchestrator client import to benchmarks. |
| packages/cli/src/sandbox/types.ts | Adds sandbox benchmark types (TTI, modes, results). |
| packages/cli/src/sandbox/table.ts | Adds CLI table rendering + JSON writer for sandbox benchmark results. |
| packages/cli/src/sandbox/staggered.ts | Adds staggered-mode sandbox benchmark runner. |
| packages/cli/src/sandbox/scoring.ts | Adds sandbox composite scoring + sorting utilities. |
| packages/cli/src/sandbox/providers.ts | Rehomes sandbox provider list under packages/cli workspace. |
| packages/cli/src/sandbox/generate-svg.ts | Updates repo-root path resolution for sandbox SVG rendering. |
| packages/cli/src/sandbox/generate-pricing-svg.ts | Updates repo-root path resolution for pricing SVG rendering. |
| packages/cli/src/sandbox/dax.ts | Adds DAX workload benchmark runner using local dax-benchmark.sh. |
| packages/cli/src/sandbox/concurrent.ts | Adds concurrent/burst-style sandbox benchmark runner. |
| packages/cli/src/sandbox/benchmark.ts | Adds core sandbox benchmark iteration logic + reuse detection probe. |
| packages/cli/src/run.ts | Updates results output directories to repo root and wires new modes. |
| packages/cli/src/merge-results.ts | Updates repo-root path resolution for merging results under workspaces. |
| packages/cli/src/ingest.ts | Updates repo-root path resolution for ingest tooling under workspaces. |
| packages/cli/src/env.ts | Updates .env resolution to repo root under workspaces. |
| packages/cli/src/browser/types.ts | Adds browser benchmark types. |
| packages/cli/src/browser/throughput-types.ts | Adds browser throughput benchmark types/constants. |
| packages/cli/src/browser/throughput-scoring.ts | Adds throughput composite scoring + sorting. |
| packages/cli/src/browser/throughput-providers.ts | Adds throughput provider configs with standardized viewport options. |
| packages/cli/src/browser/scoring.ts | Adds browser benchmark composite scoring + sorting. |
| packages/cli/src/browser/providers.ts | Adds browser provider configs. |
| packages/cli/src/browser/generate-throughput-svg.ts | Updates repo-root path resolution for throughput SVG rendering. |
| packages/cli/src/browser/generate-svg.ts | Updates repo-root path resolution for browser SVG rendering. |
| packages/cli/src/browser/benchmark.ts | Adds browser benchmark runner using Playwright CDP connection. |
| packages/cli/scripts/find-northflank-plan.ts | Adds helper script to select a suitable Northflank deployment plan. |
| packages/cli/scripts/dax-benchmark.sh | Adds local bash script for DAX clone/install/typecheck workload. |
| packages/cli/scripts/build-e2b-template.ts | Adds helper to build an E2B template with fixed resources. |
| packages/cli/scripts/backfill.ts | Updates repo-root path resolution for backfill tooling under workspaces. |
| packages/cli/package.json | Adds CLI package scripts and depends on benchmarks workspace package. |
| packages/cli/env.example | Adds example environment variables for running benchmarks locally. |
| packages/bench/vitest.config.ts | Adds Vitest config for the benchmarks SDK package. |
| packages/bench/tsup.config.ts | Adds tsup build config for publishing benchmarks SDK. |
| packages/bench/tsconfig.json | Adds package tsconfig extending repo root config. |
| packages/bench/src/reporter.ts | Adds BenchmarkReporter wrapper for best-effort telemetry/reporting. |
| packages/bench/src/metrics.ts | Adds system metrics sampler for coordinators/workers. |
| packages/bench/src/index.ts | Exports client/worker helpers, reporter, metrics, and public types. |
| packages/bench/src/tests/integration.test.ts | Adds optional live integration smoke test (env-gated). |
| packages/bench/README.md | Adds usage/API docs for the new benchmarks SDK package. |
| packages/bench/package.json | Defines publishable benchmarks SDK package metadata and scripts. |
| package.json | Removes root scripts and drops @computesdk/bench dependency in favor of workspace package. |
| .npmrc | Adds pnpm setting (shamefully-hoist=true) for workspace installs. |
| .github/workflows/storage-benchmarks.yml | Migrates storage benchmark workflow to pnpm + new paths. |
| .github/workflows/snapshot-fork-benchmarks.yml | Migrates snapshot/fork workflow to pnpm + new paths. |
| .github/workflows/scale-image.yml | Updates scale image workflow paths to packages/cli/src/scale/**. |
| .github/workflows/sandbox-tti-benchmarks.yml | Migrates sandbox TTI workflow to pnpm + new paths. |
| .github/workflows/sandbox-dax-benchmarks.yml | Migrates DAX workflow to pnpm + new paths. |
| .github/workflows/pricing-svg.yml | Migrates pricing SVG workflow to pnpm. |
| .github/workflows/browser-throughput-benchmarks.yml | Migrates browser throughput workflow to pnpm + new paths. |
| .github/workflows/browser-benchmarks.yml | Migrates browser benchmark workflow to pnpm + new paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
b059020 to
255d621
Compare
…kages/cli Port the @computesdk/bench SDK verbatim into packages/bench (published name 'benchmarks', v0.2.0) and move the CLI runner tree into packages/cli (@benchmarks/cli, private). Add pnpm-workspace.yaml, replace package-lock.json with pnpm-lock.yaml, adjust __dirname/ROOT path math in packages/cli/src to resolve to the repo root, and move the @computesdk/bench dependency to packages/cli as benchmarks: workspace:*. results/ stays at repo root.
The bundle:scale script used repo-root-relative paths that only resolved from repo root, breaking 'pnpm --filter @benchmarks/cli run bundle:scale'. Use paths relative to the package cwd so it works via workspace filter; bundle:burst-100k delegates to it. The Dockerfile keeps its own inline repo-root esbuild command for backwards compatibility.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
- Move src/ai-gateway/* to packages/cli/src/ai-gateway/* (master added this tree post-F2-cherry-pick) - Add 6 master-added bench:ai-gateway:* + generate-ai-gateway-svg scripts to packages/cli/package.json with paths rewritten to packages/cli/src/... - Empty root package.json scripts per F2 intent (bench:* / generate:* / bundle:* / update-readme all live in packages/cli/package.json now) - Adjust __dirname/.. path math in packages/cli/src/ai-gateway/generate-svg.ts (was '../..' at src/, becomes '../../..' one level deeper under packages/cli/) - Path-fix 62 cli scripts that referenced src/... instead of packages/cli/src/...
- Remove @computesdk/bench from root deps (validation contract expects it only in packages/cli/package.json as benchmarks: workspace:*) - Add benchmarks: workspace:* to packages/cli/package.json dependencies - Bump @computesdk/createos-sandbox 0.1.3 -> 0.1.4 (master updated) - Bump @computesdk/upstash 0.3.7 -> 0.3.9 (master updated) - Drop @changesets/cli from root devDeps (was lifted from packages/cli via shamefully-hoist)
2fa0d70 to
24ad21e
Compare
AI Gateway Benchmark Results
View full run · SVG available as a build artifact |
24ad21e to
4d809af
Compare
| with: | ||
| # Full history so the rebase-on-push retry below has a merge base. | ||
| fetch-depth: 0 | ||
| - uses: pnpm/action-setup@v4 |
There was a problem hiding this comment.
P2: Third-party pnpm/action-setup action is not pinned to a commit SHA
Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is compromised.
Pin the action to a full commit SHA and verify it against the official repository.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name=".github/workflows/ai-gateway-benchmarks.yml">
<violation number="1" location=".github/workflows/ai-gateway-benchmarks.yml:56">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>Uses `pnpm/action-setup@v4` with a mutable version tag instead of a commit SHA in `.github/workflows/ai-gateway-benchmarks.yml:56`.</evidence>
<recommendation>Pin the action to a full commit SHA (e.g., `pnpm/action-setup@<sha>`) and verify it against the official pnpm/action-setup repository.</recommendation>
</violation>
</file>
|
@devin can you review this? |
Review: pnpm workspace +
|
| workflow | cache: 'pnpm' |
pnpm/action-setup |
|---|---|---|
| ai-gateway-benchmarks.yml | ✅ | ✅ |
| browser-benchmarks.yml | 2 jobs | ❌ |
| browser-throughput-benchmarks.yml | 2 jobs | ❌ |
| pricing-svg.yml | ✅ | ❌ |
| sandbox-dax-benchmarks.yml | 2 jobs | ❌ |
| sandbox-tti-benchmarks.yml | 2 jobs | ❌ |
| snapshot-fork-benchmarks.yml | 2 jobs | ❌ |
| storage-benchmarks.yml | 2 jobs | ❌ |
The runner image ships npm and Yarn but not pnpm, so actions/setup-node fails at cache: 'pnpm' with Unable to locate executable file: pnpm before anything installs. None of these ran on this PR (their paths: filters don't match a workflow-only change on a PR), so green CI here isn't evidence. When re-adding, pin to a SHA — that's also what the currently-failing Superagent Security Scan is complaining about on the one workflow that does have it.
3. 🔴 benchsdk is never built, so a fresh clone can't run or typecheck the scale runners
packages/benchsdk has build/clean/dev but no prepare/postinstall, so after pnpm install there is no packages/benchsdk/dist, and main/types point into it. Straight from a clean install:
$ pnpm run typecheck
benchmarks/scale/scripts/start.ts(46,39): error TS2307: Cannot find module 'benchsdk' …
benchmarks/scale/scripts/watch.ts(31,58): error TS2307: …
benchmarks/scale/sdk-coordinator.ts(5,83): error TS2307: …
(+18 cascading implicit-any errors) → 22 errors
$ pnpm --filter benchsdk run build && pnpm run typecheck
benchmarks/scripts/build-e2b-template.ts(2,46): error TS2307: Cannot find module 'e2b' → 1 error
Same for runtime: bench:scale:local / :start / :watch / :aggregate all import benchsdk under tsx. The Docker image is fine only because it explicitly runs pnpm --filter benchsdk run build. Adding "prepare": "tsup" to packages/benchsdk/package.json fixes both.
(That last e2b error is newly surfaced rather than newly introduced — the new root tsconfig.json adds benchmarks/scripts/** to include, and scripts/build-e2b-template.ts imports e2b, which isn't a dependency. Worth adding the dep or excluding the file, since the new typecheck script can't pass as-is.)
4. 🔴 The rebase dropped five scripts that exist on master
master's package.json has these; the branch's rewritten script block doesn't:
bench:ai-gateway:llmgateway
bench:ai-gateway:pydantic
bench:browser:tilion
bench:browser-throughput:tilion
bench:opencomputer
All five providers still exist in code (benchmarks/ai-gateway/providers.ts:54,76, benchmarks/browser/providers.ts:77, benchmarks/browser/throughput-providers.ts:103, benchmarks/sandbox/providers.ts:155) and AI_GATEWAYS.md:117-118 documents the two gateway ones.
Should fix before merge
packages/benchsdk/README.mdis still the@computesdk/benchREADME — title,npm install @computesdk/bench, and every import example. It's infiles, so the publishedbenchsdktarball would tell users to install a different package.package.jsonalso has a stalerepository.directory: "packages/bench".benchmarks/scale/Dockerfile.norunis dead — it still doesCOPY package.json package-lock.json ./,npm ci,COPY src ./src; the lockfile andsrc/are both gone in this PR..envlocation changed silently.benchmarks/src/env.tsloads../.env, i.e.benchmarks/.envnow, andenv.examplemoved tobenchmarks/.env.example.CONTRIBUTING.md:72andMETHODOLOGY.md:305still saycp env.example .envat the root — existing local.envfiles stop being picked up with no error. Same staleness inAI_GATEWAYS.md(src/ai-gateway/…,npx tsx src/run.ts) and.claude/skills/add-sandbox-provider/SKILL.md(links tosrc/sandbox/providers.ts,npm install), plus all thenpm runsnippets across the docs.- The new package has no CI. Nothing builds, typechecks, tests or publishes
benchsdkon PRs — for a package that's meant to go to npm, a small workflow runningpnpm --filter benchsdk run build && test && typecheckseems worth it. Also"lint": "eslint"with no eslint config anywhere in the repo will fail if anyone runs it, and the devDeps are skewed from the root (vitest 1.x, eslint 8 EOL, TS ^5 vs root TS ^6).
Nits
__BENCH_VERSION__isdefined intsup.config.tsbut never referenced insrc/.- pnpm blocks postinstall scripts by default — installs print
Ignored build scripts: esbuild, protobufjs, bufferutil, cbor-extract, …. Nothing broke in my testing, but an explicitpnpm.onlyBuiltDependenciesin the rootpackage.jsonwould make that deliberate rather than a warning everyone learns to ignore. - Workflows still shell out to
npx tsx …in a pnpm repo;pnpm exec tsxis more consistent (and avoids npx's fallback-to-registry behaviour). benchsdkis unscoped and currently unpublished on npm, while everything else here is@computesdk/*. If unscoped is deliberate, fine — otherwise@computesdk/benchsdkis more consistent and not squattable. Either way, the existing@computesdk/bench@0.1.9probably wants a deprecation pointing at the new name.
4d809af to
abe635f
Compare
Re-review of
|
| # | Was | Now |
|---|---|---|
| 1 | 661 results nested in results/results/ |
git diff --merge-base master -- results/ is empty — history restored byte-for-byte. pnpm run generate-svg writes all four SVGs and leaves the tree clean |
| 2 | pnpm/action-setup missing in 7 workflows |
present in all 8, immediately before actions/setup-node in every job |
| 3 | benchsdk never built |
"prepare": "tsup" → dist/ exists straight after install; pnpm run typecheck is now completely clean (e2b added as a dep, so the old TS2307 is gone too); 27 benchsdk tests pass |
| 4 | 5 scripts lost in the rebase | all restored — typecheck is the only script that differs from master now |
The smaller items are done too: benchsdk README and repository.directory renamed, Dockerfile.norun deleted, docs moved to pnpm + benchmarks/.env.example.
🔴 One new regression: the prepare script broke the scale image build
Build and push scale image is now failing on this PR:
#12 2.031 packages/benchsdk prepare$ tsup
#12 2.196 packages/benchsdk prepare: No input files, try "tsup <your-file>" instead
#12 2.200 packages/benchsdk prepare: Failed
ERROR: process "/bin/sh -c pnpm install --frozen-lockfile" did not complete successfully: exit code: 1
benchmarks/scale/Dockerfile copies only the manifests first (for layer caching), so pnpm install now runs prepare → tsup before packages/benchsdk/src has been copied in:
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY packages/benchsdk/package.json packages/benchsdk/
COPY tsconfig.json ./
RUN pnpm install --frozen-lockfile # ← prepare runs here, no src yet
COPY benchmarks ./benchmarks
COPY packages/benchsdk ./packages/benchsdk
RUN pnpm --filter benchsdk run build # ← the real build, already hereSmallest fix, which I built locally to confirm (docker build -f benchmarks/scale/Dockerfile . fails on abe635f, succeeds with this one-word change):
-RUN pnpm install --frozen-lockfile
+RUN pnpm install --frozen-lockfile --ignore-scriptsThe explicit pnpm --filter benchsdk run build after the COPY still produces dist/, so nothing else changes.
Still open (all minor, none blocking)
- Superagent Security Scan is still red:
pnpm/action-setup@v4is unpinned, now in 14 places. Pinning to a SHA (pnpm/action-setup@<sha> # v4.x.x) clears it. .claude/skills/add-sandbox-provider/SKILL.mdwasn't updated — it still tells the next agent to editenv.example,src/sandbox/providers.ts,src/sandbox/generate-svg.ts,src/sandbox/benchmark.tsand runnpm install. Since that skill is the documented path for adding providers, it'll produce wrong edits on the next provider PR.CONTRIBUTING.md:71andMETHODOLOGY.md:304still saynpm installdirectly above thepnpm run …blocks;AI_GATEWAYS.md:128still citesenv.example.benchsdkstill has"lint": "eslint"with no eslint config anywhere in the repo, and no workflow builds/tests/publishes it on PRs.__BENCH_VERSION__is stilldefined intsup.config.tsand never referenced.e2b: ^2.35.0landed out of alphabetical order (right after@computesdk/e2b) independencies.
Fix the Dockerfile line and this looks good to merge from my side.
- Rename SDK from packages/bench to packages/benchsdk (npm name: benchsdk) - Move all runners, src, scripts, results, SVGs, pricing into benchmarks/ - Update all __dirname path math for new directory depth - Rewrite SDK imports from @computesdk/bench to benchsdk - Update 8 CI workflows: benchmarks/ paths + pnpm commands - Fix relative imports: ./sandbox/ -> ../sandbox/, ../util/ -> ../src/util/ - Delete packages/cli (scripts consolidated into root package.json) - Add pnpm/action-setup to ai-gateway workflow - Rebase onto origin/master: merge master's new providers (opencomputer, tilion, llmgateway, pydantic) and dep bumps (beam, superserve, tenki)
abe635f to
ec8a8c4
Compare
Re-review of
|
Storage Benchmark Results1MB Files
View full run · SVGs available as build artifacts |
Snapshot/Fork Benchmark Resultssmall dataset
|
Browser Benchmark Results
View full run · SVG available as build artifact |
Browser Throughput Benchmark Results
View full run · SVG available as build artifact |
Sandbox Benchmark ResultsSequential
Staggered
Burst
View full run · SVGs available as build artifacts |
Sandbox Dax Benchmark Results
|
Resolves the conflicts surfaced after master advanced with the pnpm-workspace + SDK-port restructuring (#207). Conflicts resolved: - 3 file-location rename conflicts (scripts/* -> benchmarks/scripts/*): auto-detected by git's rename detection; the 3 new files (gen-provider-vars.ts, provider-vars.json, write-vault-envdef.mjs) landed at benchmarks/scripts/ alongside the other auto-renamed scripts. - 4 workflow content conflicts (browser, sandbox-dax, sandbox-tti, storage): kept the new vault-load steps (Load ingest secrets from Namespace vault) over master's GH-secret env blocks, AND adopted master's pnpm + benchmarks/src/ + benchmarks/scripts/ paths. - scripts/write-vault-envdef.mjs paths updated in all 7 workflows to benchmarks/scripts/write-vault-envdef.mjs. - gen-provider-vars.ts: imports updated to resolve via the new benchmarks/{mode}/providers.ts layout; doc comment updated. After re-resolve YAML still parses cleanly via js-yaml. Helper smoke-tests produce the same envdefs as before. Real-CLI roundtrip on hyperbrowser still resolves a 31-char secret end-to-end.
Summary
Restructures the benchmarks repo to extract the bench SDK from the computesdk monorepo into a standalone publishable package, and consolidates all benchmark runners under a
benchmarks/wrapper.SDK (publishable)
@computesdk/benchin the computesdk monorepopackages/benchsdk/with npm namebenchsdk(unscoped)0.2.0, ready for npm publishRunners (in-house, under
benchmarks/)benchmarks/sandbox/— sandbox TTI benchmarks (sequential, staggered, burst, dax)benchmarks/browser/— browser + throughput benchmarksbenchmarks/storage/— storage + snapshot-fork benchmarksbenchmarks/ai-gateway/— AI gateway benchmarksbenchmarks/scale/— scale coordinator + Docker imagebenchmarks/src/— shared helpers (run.ts, ingest.ts, merge-results.ts, env.ts, update-readme.ts, util/)benchmarks/scripts/— operational scripts (backfill, build-e2b-template, dax-benchmark.sh, etc.)Repo-level changes
pnpm-workspace.yamldeclarespackages/benchsdk)package.jsonholds all runner scripts withbenchmarks/-prefixed pathspnpm install --frozen-lockfile,pnpm run bench)@computesdk/benchdependency replaced with localbenchsdk: workspace:*@computesdk/benchtobenchsdk.env.examplemoved tobenchmarks/.env.exampleWhat stays at repo root
results/,*.svg,pricing.json— generated benchmark artifactsREADME.md,LICENSE,CONTRIBUTING.md,METHODOLOGY.md, etc. — repo docspackages/benchsdk/— the SDK packageRebased onto master
Includes all of master's latest providers (
opencomputer,tilion,llmgateway,pydantic) and dependency bumps (beam,superserve,tenki,@codesandbox/sdk).Commits
fd90611feat: convert benchmarks to pnpm workspaces with packages/bench + packages/cli7ea9222fix(cli): make bundle:scale paths relative to packages/cli cwd66da0ddUpdate package.json (packageManager: pnpm@10.33.0)9123781chore(rebase): followup ai-gateway move after conflict resolution2cfbc09chore(rebase): sync pnpm-lock.yaml with post-rebase package.json4d809affeat: restructure to benchsdk package + benchmarks/ wrapper