Skip to content

docs: add benchsdk authoring guide, examples, and TaskError bundle fix - #366

Merged
dtice25 merged 9 commits into
masterfrom
devin/benchsdk-docs-examples
Aug 26, 2026
Merged

dtice25 merged 9 commits into
masterfrom
devin/benchsdk-docs-examples

Conversation

@dtice25

@dtice25 dtice25 commented Aug 24, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR makes it clear how to write and run benchmarks with @benchsdk/runner, and adds runnable examples for every major benchsdk capability.

  • Adds WRITING_BENCHMARKS.md, a single authoring guide covering create-bench, defineBenchmarkConfig, defineTask, step, measure, scoring, participants, CLI flags, and the platform.
  • Adds examples/ with 10 self-contained *.bench.ts files (01-hello through 10-shared-run) that exercise shapes, phases, round-robin, scoring, custom CLI flags, error handling, env-gated participants, onScore/onComplete, and shared runs. All run offline with --dry-run.
  • Cross-links the guide and examples from the root README.md, llms.txt, packages/benchsdk-runner/README.md, and packages/create-bench/README.md.
  • Fixes a cross-bundle TaskError identity issue: the bench CJS binary and ESM-imported benchmarks load separate class copies, so instanceof TaskError and error-code extraction failed. getErrorCode now prefers error.code when present, and the runner uses duck-typing (error.name === 'TaskError') to preserve code, data, and steps regardless of which bundle constructed the error.

Link to Devin session: https://app.devin.ai/sessions/5673cc677c724d538eae9e90758ac8a8
Requested by: @dtice25


Open in Devin Review

…Error cross-bundle recognition

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 24, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Resolved.

dtice25 and others added 6 commits August 25, 2026 18:19
…ome, and unified bench CLI

- Expand WRITING_BENCHMARKS.md with log levels, step output capture,
  isStepOutcome/captureOutput, and platform query commands.
- Add examples/11-logging.bench.ts demonstrating structured logging,
  step output capture, and captureOutput: false.
- Update existing examples to use leveled log calls.
- Cross-link .agents/skills/benchsdk-cli/SKILL.md from READMEs and llms.txt.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The benchmarks platform only accepts BENCHMARKS_PLATFORM_API_KEY. Remove the
COMPUTESDK_API_KEY env fallback from benchsdk-api, benchsdk-cli,
benchsdk-runner, scale scripts, and docs; update the public-api contract test
and integration guard accordingly.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The gateway-provider concept is no longer relevant. Remove the
GATEWAY_PROVIDERS = ['render'] constant and the '*' suffix in
formatProviderName, drop the orchestrator footnote, and remove the skill
bullet that referenced it. Also rename the WRITING_BENCHMARKS groupBy
example variable from gatewayProviders to providers to avoid confusion.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Restore benchmarks/scale/* to the state before the COMPUTESDK_API_KEY
cleanup so the scale invitational scripts are not modified on this PR.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@dtice25
dtice25 merged commit 3095712 into master Aug 26, 2026
6 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 createos 98.6 0.13s 0.14s 0.14s 10/10
2 daytona 97.3 0.20s 0.38s 0.38s 10/10
3 arker 96.8 0.32s 0.32s 0.32s 10/10
4 archil 96.5 0.34s 0.36s 0.36s 10/10
5 declaw 96.2 0.36s 0.41s 0.41s 10/10
6 mosaic 96.1 0.37s 0.42s 0.42s 10/10
7 blaxel 94.8 0.50s 0.56s 0.56s 10/10
8 superserve 93.2 0.66s 0.70s 0.70s 10/10
9 upstash 92.0 0.77s 0.84s 0.84s 10/10
10 vercel 91.8 0.54s 1.22s 1.22s 10/10
11 runloop 91.7 0.78s 0.90s 0.90s 10/10
12 modal 90.2 0.78s 1.28s 1.28s 10/10
13 tenki 88.4 1.12s 1.22s 1.22s 10/10
14 e2b 88.3 1.12s 1.25s 1.25s 10/10
15 beam 85.3 1.34s 1.66s 1.66s 10/10
16 tensorlake 83.0 1.20s 2.45s 2.45s 10/10
17 sail 82.7 1.66s 1.85s 1.85s 10/10
18 cloud-run 76.7 1.75s 3.20s 3.20s 10/10
19 cloudflare 65.8 2.80s 4.35s 4.35s 10/10
20 codesandbox 64.7 3.17s 4.07s 4.07s 10/10
21 run-cloud 25.6 6.40s 8.99s 8.99s 10/10
22 microsandbox 18.6 6.89s 12.44s 12.44s 10/10
23 opencomputer 11.0 8.38s 9.68s 9.68s 10/10
24 hopx 0.0 0.00s 0.00s 0.00s 0/10
25 isorun 0.0 0.00s 0.00s 0.00s 0/10
26 lightning 0.0 0.00s 0.00s 0.00s 0/10
27 miosa 0.0 10.37s 20.31s 20.31s 10/10
28 northflank 0.0 0.00s 0.00s 0.00s 0/10
29 sandbox0 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
arker 7/7 80.77s 5.35s 27.29s 11.19s 32.18s 1/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 7/7 45.46s 1.24s 1.54s 9.98s 24.39s 1/1 OK
cloud-run 2/7 1.24s 0.24s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 65.09s 3.55s 15.07s 11.28s 26.72s 1/1 OK
daytona 7/7 70.47s 4.82s 1.73s 13.41s 36.33s 1/1 OK
declaw 7/7 169.74s 6.72s 4.03s 26.41s 119.66s 1/1 OK
e2b 6/7 29.23s 8.30s 3.83s 9.82s 0.00s 0/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
microsandbox 4/7 7.61s 4.40s 0.00s 0.00s 0.00s 0/1 OK
miosa 2/7 3.37s 2.94s 0.00s 0.00s 0.00s 0/1 OK
modal 7/7 111.90s 8.31s 23.04s 13.44s 59.51s 1/1 OK
mosaic 7/7 62.13s 1.19s 3.11s 14.33s 31.73s 1/1 OK
namespace 7/7 58.29s 5.64s 2.29s 11.40s 29.92s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
run-cloud -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 6/7 124.29s 28.42s 8.31s 27.68s 0.00s 0/1 OK
sail 7/7 93.91s 6.64s 3.89s 19.07s 52.07s 1/1 OK
sandbox0 -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
superserve 7/7 106.23s 11.81s 5.54s 20.00s 64.07s 1/1 OK
tenki 7/7 56.48s 4.86s 3.99s 17.38s 23.59s 1/1 OK
tensorlake 7/7 98.33s 20.18s 5.90s 17.94s 45.92s 1/1 OK
upstash 7/7 51.05s 3.83s 1.77s 10.02s 25.13s 1/1 OK
vercel 7/7 101.61s 13.15s 32.89s 12.33s 36.27s 1/1 OK

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant