Skip to content

refactor: remove benchmarkKind from benchsdk and benchmarks - #302

Merged
dtice25 merged 2 commits into
masterfrom
devin/remove-benchmark-kind
Aug 11, 2026
Merged

dtice25 merged 2 commits into
masterfrom
devin/remove-benchmark-kind

Conversation

@dtice25

@dtice25 dtice25 commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Removes the benchmarkKind / kind concept entirely from the benchmark SDK, runner, and legacy ingestion flow.

  • benchsdk types: BenchmarkResource, UpsertBenchmarkInput, and UpdateBenchmarkInput no longer accept kind; BenchmarkResultsOverview and BenchmarkRunResults no longer pick it.
  • benchsdk-runner: BenchmarkConfig.benchmarkKind and BenchmarkShape.kind removed; applyShape and the platform upsertBenchmark call no longer pass a kind.
  • Legacy benchmarks/src/ingest.ts no longer derives or sends kind.
  • All .bench.ts files and READMEs updated to drop benchmarkKind.
  • Test fixtures (public-api.contract.test.ts, client.test.ts, integration.test.ts, runner.test.ts, bench-config.test.ts) updated.

Verified with pnpm typecheck, pnpm --filter @benchsdk/client test, pnpm --filter @benchsdk/runner test, and pnpm -r --filter "./packages/**" build.

Link to Devin session: https://app.devin.ai/sessions/4f342077d5e64ce69e8b771e22845ec5
Requested by: @dtice25


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Open in Devin Review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25 dtice25 self-assigned this Aug 11, 2026
@open-cla

open-cla Bot commented Aug 11, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25
dtice25 merged commit 19582a3 into master Aug 11, 2026
7 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.6 0.02s 365.3 Mbps 0.07s 10/10
2 Archil 95.3 0.05s 153.3 Mbps 0.19s 10/10
3 Vercel Blob 95.2 0.06s 146.0 Mbps 0.23s 10/10
4 Tigris 94.8 0.08s 107.7 Mbps 0.25s 10/10
5 Tensorlake 94.8 0.05s 185.7 Mbps 0.59s 10/10
6 AWS S3 94.8 0.10s 84.0 Mbps 0.19s 10/10
7 Neon 94.7 0.12s 67.9 Mbps 0.28s 10/10
8 Cloudflare R2 94.5 0.12s 70.0 Mbps 0.29s 10/10
9 Google Cloud Storage 0.0 0.00s 0.0 Mbps 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.3 0.07s 0.07s 0.07s 10/10
2 createos 99.2 0.08s 0.09s 0.09s 10/10
3 declaw 98.4 0.08s 0.28s 0.28s 10/10
4 arker 98.3 0.16s 0.18s 0.18s 10/10
5 northflank 97.4 0.26s 0.27s 0.27s 10/10
6 lightning 97.2 0.19s 0.41s 0.41s 10/10
7 blaxel 97.0 0.30s 0.31s 0.31s 10/10
8 archil 96.1 0.30s 0.52s 0.52s 10/10
9 opencomputer 96.0 0.37s 0.45s 0.45s 10/10
10 superserve 95.0 0.49s 0.51s 0.51s 10/10
11 modal 94.3 0.53s 0.61s 0.61s 10/10
12 vercel 93.5 0.41s 1.00s 1.00s 10/10
13 runloop 93.3 0.66s 0.69s 0.69s 10/10
14 tensorlake 89.7 0.84s 1.31s 1.31s 10/10
15 upstash 89.5 0.74s 1.51s 1.51s 10/10
16 mosaic 88.4 1.12s 1.22s 1.22s 10/10
17 e2b 88.3 1.10s 1.28s 1.28s 10/10
18 cloud-run 85.3 1.37s 1.62s 1.62s 10/10
19 sandbox0 81.8 1.77s 1.88s 1.88s 10/10
20 beam 81.5 1.85s 1.87s 1.87s 10/10
21 sail 80.1 1.98s 1.99s 1.99s 10/10
22 cloudflare 79.9 1.84s 2.26s 2.26s 10/10
23 codesandbox 64.3 3.23s 4.07s 4.07s 10/10
24 run-cloud 64.3 3.48s 3.71s 3.71s 10/10
25 daytona 51.0 1.51s 22.68s 22.68s 10/10
26 hopx 0.0 0.00s 0.00s 0.00s 0/10
27 tenki 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tensorlake 99.7 0.03s 0.28s 0.32s 0.16s 1/1
2 Tigris 99.6 0.22s 0.24s 0.41s 0.10s 1/1
3 Cloudflare R2 83.4 8.82s 9.25s 20.95s 0.30s 1/1
4 Archil 0.0 0.00s 0.00s 0.00s 0.00s 0/1
5 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1
6 Azure-blob 0.0 0.00s 0.00s 0.00s 0.00s 0/1
7 Neon 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 98.1 0.04s 0.05s 0.10s 0.04s 0.22s 10/10
2 Tilion 96.7 0.11s 0.04s 0.09s 0.04s 0.29s 10/10
3 Browseruse 95.9 0.16s 0.11s 0.08s 0.04s 0.42s 10/10
4 Browserbase 93.3 0.16s 0.30s 0.12s 0.13s 0.78s 10/10
5 Hyperbrowser 91.9 0.25s 0.14s 0.12s 0.09s 0.62s 10/10
6 Notte 89.1 0.25s 0.57s 0.21s 0.11s 1.30s 10/10
7 Steel 84.5 0.28s 0.93s 0.14s 0.16s 1.62s 10/10

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 73.4 4.27/s 2.34s 2.42s 275ms 3/3
2 Browserbase 66.5 3.02/s 3.32s 3.95s 335ms 3/3
3 Tilion 66.2 3.00/s 3.33s 4.32s 359ms 3/3
4 Browseruse 64.0 2.58/s 3.88s 4.27s 389ms 3/3
5 Hyperbrowser 55.5 1.63/s 6.12s 8.53s 568ms 3/3
6 Steel 51.7 1.20/s 8.35s 8.73s 727ms 3/3
7 Notte 46.9 0.95/s 10.52s 11.29s 1329ms 3/3

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Anthropic (direct) 89.7 512ms 486ms 57.3 20/20
2 Pydantic AI Gateway 89.6 593ms 512ms 57.4 20/20
3 Cloudflare AI Gateway 89.0 511ms 508ms 57.5 20/20
4 OpenRouter 88.9 511ms 509ms 55.5 20/20
5 Vercel AI Gateway 88.3 691ms 567ms 53.3 20/20
6 Concentrate AI 88.0 852ms 642ms 49.6 20/20
7 LLM Gateway 86.4 658ms 566ms 53.1 20/20

View full run · SVG available as a build artifact

kisernl added a commit that referenced this pull request Aug 11, 2026
Master removed benchmarkKind from the SDK and every bench file (#302);
this benchmark was branched before that and was the last occurrence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant