Repository navigation
docs: add benchsdk authoring guide, examples, and TaskError bundle fix - #366
Merged
Merged
Conversation
…Error cross-bundle recognition Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor License AgreementAll contributors are covered by a CLA. |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
Resolved. |
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
Resolved. |
…ome, and unified bench CLI - Expand WRITING_BENCHMARKS.md with log levels, step output capture, isStepOutcome/captureOutput, and platform query commands. - Add examples/11-logging.bench.ts demonstrating structured logging, step output capture, and captureOutput: false. - Update existing examples to use leveled log calls. - Cross-link .agents/skills/benchsdk-cli/SKILL.md from READMEs and llms.txt. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The benchmarks platform only accepts BENCHMARKS_PLATFORM_API_KEY. Remove the COMPUTESDK_API_KEY env fallback from benchsdk-api, benchsdk-cli, benchsdk-runner, scale scripts, and docs; update the public-api contract test and integration guard accordingly. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The gateway-provider concept is no longer relevant. Remove the GATEWAY_PROVIDERS = ['render'] constant and the '*' suffix in formatProviderName, drop the orchestrator footnote, and remove the skill bullet that referenced it. Also rename the WRITING_BENCHMARKS groupBy example variable from gatewayProviders to providers to avoid confusion. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Restore benchmarks/scale/* to the state before the COMPUTESDK_API_KEY cleanup so the scale invitational scripts are not modified on this PR. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Sandbox Benchmark ResultsBurst
View full run · SVGs available as build artifacts |
Contributor
Sandbox Dax Benchmark Results
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes it clear how to write and run benchmarks with
@benchsdk/runner, and adds runnable examples for every major benchsdk capability.WRITING_BENCHMARKS.md, a single authoring guide coveringcreate-bench,defineBenchmarkConfig,defineTask,step,measure, scoring, participants, CLI flags, and the platform.examples/with 10 self-contained*.bench.tsfiles (01-hello through 10-shared-run) that exercise shapes, phases, round-robin, scoring, custom CLI flags, error handling, env-gated participants,onScore/onComplete, and shared runs. All run offline with--dry-run.README.md,llms.txt,packages/benchsdk-runner/README.md, andpackages/create-bench/README.md.TaskErroridentity issue: thebenchCJS binary and ESM-imported benchmarks load separate class copies, soinstanceof TaskErrorand error-code extraction failed.getErrorCodenow preferserror.codewhen present, and the runner uses duck-typing (error.name === 'TaskError') to preservecode,data, andstepsregardless of which bundle constructed the error.Link to Devin session: https://app.devin.ai/sessions/5673cc677c724d538eae9e90758ac8a8
Requested by: @dtice25