Skip to content

refactor(sandbox): retire sequential/staggered TTI shapes, keep burst… - #303

Merged
dtice25 merged 5 commits into
masterfrom
feat/update-naming-schema
Aug 11, 2026
Merged

dtice25 merged 5 commits into
masterfrom
feat/update-naming-schema

Conversation

@dtice25

@dtice25 dtice25 commented Aug 11, 2026 •

Copy link
Copy Markdown
Collaborator

… only

  • Remove sequential/staggered modes from sandbox-tti-benchmarks.yml workflow input and bench step logic; always run burst shape with user-specified concurrency.
  • Update tti.bench.ts to drop shapes config and legacyShape helper; hardcode burst resultsDir and 'concurrent' mode for legacy JSON.
  • Rename benchmark slugs to drop '-local' suffix: sandbox-tti, sandbox-dax, browser-lifecycle, browser-throughput, ai-gateway-latency, storage-lifecycle

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Open in Devin Review

… only

- Remove sequential/staggered modes from sandbox-tti-benchmarks.yml workflow input and bench step logic; always run burst shape with user-specified concurrency.
- Update tti.bench.ts to drop shapes config and legacyShape helper; hardcode burst resultsDir and 'concurrent' mode for legacy JSON.
- Rename benchmark slugs to drop '-local' suffix: sandbox-tti, sandbox-dax, browser-lifecycle, browser-throughput, ai-gateway-latency, storage-lifecycle
@open-cla

open-cla Bot commented Aug 11, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 3 additional findings in Devin Review.

Open in Devin Review

Comment thread benchmarks/src/ingest.ts Resolved
dtice25 and others added 4 commits August 11, 2026 16:08
…ript

- Delete benchmarks/src/ingest.ts since bench steps now stream events directly to platform orchestrator endpoints.
- Remove collect-job "Ingest results to platform" steps from browser-benchmarks.yml and storage-benchmarks.yml workflows.
- Update dry_run input descriptions across all workflows to reflect that ingestion is no longer a separate step: 'Run without committing results' instead of 'Run without ingesting or committing results'.
…tial

Sequential/staggered modes are retired; burst is now the only active TTI shape.
Update generate-svg.ts to write the backward-compat root results.svg from burst
data instead of sequential.
The platform now auto-generates run names, so drop the explicit 'name' field from createRun/updateRun calls and assert run.name is present instead of checking a specific value.
@dtice25
dtice25 merged commit aff1274 into master Aug 11, 2026
8 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.7 0.03s 0.06s 0.09s 0.04s 0.22s 10/10
2 Tilion 96.7 0.10s 0.04s 0.08s 0.04s 0.27s 10/10
3 Browseruse 95.3 0.16s 0.12s 0.10s 0.04s 0.45s 10/10
4 Browserbase 93.2 0.16s 0.29s 0.11s 0.14s 0.72s 10/10
5 Hyperbrowser 89.7 0.27s 0.16s 0.12s 0.09s 0.70s 10/10
6 Notte 88.5 0.22s 0.51s 0.21s 0.09s 1.17s 10/10
7 Steel 80.2 0.30s 0.84s 0.12s 0.17s 1.56s 10/10

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tensorlake 99.7 0.03s 0.28s 0.31s 0.12s 1/1
2 Tigris 99.1 0.12s 1.26s 0.25s 0.10s 1/1
3 Cloudflare R2 83.2 9.01s 10.76s 17.84s 0.26s 1/1
4 Archil 0.0 0.00s 0.00s 0.00s 0.00s 0/1
5 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1
6 Azure-blob 0.0 0.00s 0.00s 0.00s 0.00s 0/1
7 Neon 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.0 0.03s 257.4 Mbps 0.09s 10/10
2 Archil 95.0 0.08s 104.3 Mbps 0.18s 10/10
3 AWS S3 94.8 0.10s 87.2 Mbps 0.19s 10/10
4 Tensorlake 94.7 0.06s 138.2 Mbps 0.55s 10/10
5 Tigris 94.6 0.05s 154.9 Mbps 0.30s 10/10
6 Vercel Blob 94.4 0.09s 94.5 Mbps 0.30s 10/10
7 Cloudflare R2 94.4 0.17s 52.3 Mbps 0.30s 10/10
8 Neon 94.4 0.12s 67.4 Mbps 0.29s 10/10
9 Google Cloud Storage 0.0 0.00s 0.0 Mbps 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 44.5 4.03/s 2.48s 10.64s 270ms 2/3
2 Browserbase 42.3 3.37/s 2.97s 11.07s 332ms 2/3
3 Tilion 41.6 3.10/s 3.22s 10.73s 236ms 2/3
4 Browseruse 38.5 2.31/s 4.33s 11.36s 527ms 2/3
5 Hyperbrowser 35.9 1.65/s 6.05s 11.21s 548ms 2/3
6 Steel 32.3 1.10/s 9.11s 12.03s 715ms 2/3
7 Notte 31.9 1.10/s 9.06s 12.62s 1044ms 2/3

View full run · SVG available as build artifact

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.2 0.06s 0.11s 0.11s 10/10
2 declaw 97.8 0.17s 0.28s 0.28s 10/10
3 arker 97.7 0.21s 0.26s 0.26s 10/10
4 createos 97.2 0.28s 0.29s 0.29s 10/10
5 northflank 97.1 0.26s 0.33s 0.33s 10/10
6 lightning 96.1 0.33s 0.47s 0.47s 10/10
7 tensorlake 95.1 0.49s 0.50s 0.50s 10/10
8 opencomputer 94.9 0.49s 0.55s 0.55s 10/10
9 modal 94.3 0.53s 0.62s 0.62s 10/10
10 upstash 93.2 0.64s 0.75s 0.75s 10/10
11 blaxel 92.4 0.75s 0.78s 0.78s 10/10
12 mosaic 92.1 0.69s 0.93s 0.93s 10/10
13 archil 90.7 0.93s 0.94s 0.94s 10/10
14 beam 90.3 0.97s 0.98s 0.98s 10/10
15 superserve 89.9 0.61s 1.61s 1.61s 10/10
16 e2b 89.4 0.93s 1.24s 1.24s 10/10
17 daytona 87.2 1.19s 1.41s 1.41s 10/10
18 sandbox0 83.9 1.50s 1.78s 1.78s 10/10
19 sail 80.5 1.95s 1.96s 1.96s 10/10
20 runloop 78.0 0.90s 4.14s 4.14s 10/10
21 cloudflare 77.9 2.05s 2.45s 2.45s 10/10
22 vercel 65.5 0.73s 7.55s 7.55s 10/10
23 codesandbox 61.7 3.61s 4.17s 4.17s 10/10
24 cloud-run 57.4 3.90s 4.79s 4.79s 10/10
25 run-cloud 31.2 5.82s 8.47s 8.47s 10/10
26 hopx 0.0 0.00s 0.00s 0.00s 0/10
27 tenki 0.0 0.00s 0.00s 0.00s 0/10

View full run · SVGs available as build artifacts

@github-actions

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
arker 7/7 59.75s 4.11s 2.53s 11.99s 34.99s 1/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 7/7 45.17s 1.22s 1.47s 9.78s 24.48s 1/1 OK
cloud-run 2/7 1.27s 0.24s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 7/7 50.29s 2.81s 2.20s 10.74s 25.88s 1/1 OK
daytona 7/7 71.08s 5.58s 1.94s 13.52s 35.01s 1/1 OK
declaw 7/7 68.25s 5.42s 2.19s 11.92s 41.13s 1/1 OK
e2b 7/7 70.21s 8.25s 4.48s 8.49s 41.15s 1/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun 7/7 93.67s 0.93s 1.39s 10.44s 25.34s 1/1 OK
lightning 7/7 39.29s 3.74s 3.34s 12.62s 13.98s 1/1 OK
modal 7/7 88.92s 9.34s 2.55s 13.44s 56.07s 1/1 OK
mosaic 7/7 77.25s 14.64s 4.84s 14.11s 31.31s 1/1 OK
namespace 7/7 43.83s 5.09s 4.64s 7.55s 19.40s 1/1 OK
northflank -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
opencomputer 6/7 25.19s 3.78s 2.15s 17.29s 0.00s 0/1 OK
run-cloud -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 2/7 118.42s 0.00s 0.00s 26.40s 54.73s 1/1 OK
sail 7/7 79.84s 1.96s 4.02s 13.56s 52.86s 1/1 OK
sandbox0 7/7 201.73s 5.80s 7.59s 73.73s 82.95s 1/1 OK
superserve 7/7 95.27s 9.03s 5.26s 21.83s 55.44s 1/1 OK
tenki -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
tensorlake 7/7 66.60s 14.19s 1.96s 10.41s 33.77s 1/1 OK
upstash 7/7 52.89s 5.78s 1.89s 9.47s 25.34s 1/1 OK
vercel 7/7 74.93s 17.80s 2.10s 11.77s 36.74s 1/1 OK

View full run

@github-actions

Copy link
Copy Markdown
Contributor

AI Gateway Benchmark Results

# Gateway Score Cold E2E Warm TTFT Tok/sec Status
1 Anthropic (direct) 89.6 513ms 464ms 59.0 20/20
2 LLM Gateway 89.4 651ms 541ms 54.2 20/20
3 Cloudflare AI Gateway 89.4 515ms 485ms 56.0 20/20
4 OpenRouter 89.2 508ms 504ms 54.7 20/20
5 Vercel AI Gateway 88.8 717ms 561ms 53.4 20/20
6 Concentrate AI 88.5 771ms 669ms 47.8 20/20
7 Pydantic AI Gateway 88.0 625ms 490ms 54.4 20/20

View full run · SVG available as a build artifact

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant