Skip to content

feat(benchmarks): add reproducible generation benchmark - #330

Open
vetertann wants to merge 1 commit into
toon-format:mainfrom
vetertann:codex/typescript-generation-benchmark
Open

vetertann wants to merge 1 commit into
toon-format:mainfrom
vetertann:codex/typescript-generation-benchmark

Conversation

@vetertann

Copy link
Copy Markdown

Linked Issue

Closes #207
Supersedes #239

Description

This PR ports the structured generation benchmark to TypeScript and integrates it into the existing benchmarks workspace. It ships the reproducible harness, gold fixtures, and the original 21-model result set together.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Changes Made

  • Added a TypeScript benchmark runner for plain JSON, JSON object mode, and TOON generation.
  • Added strict schemas, manual validation, canonicalization, and up to two repair attempts.
  • Uses the local TypeScript TOON encoder/decoder without Python or CLI subprocesses.
  • Added fixture generation and committed JSON/TOON gold fixtures for all four cases.
  • Added the historical 21-model raw and aggregate result CSVs.
  • Added Nebius configuration, model/run selection, checkpoints, request timeouts, and usage accounting.
  • Documented setup, methodology, commands, and the published baseline results.

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected: N/A (benchmark tooling)
  • Spec version: N/A

Testing

  • All existing tests pass
  • Added new tests for validation, canonicalization, wrapper extraction, and result aggregation
  • TypeScript typecheck passes
  • ESLint passes
  • Full suite passes (644 tests)
  • End-to-end Nebius smoke run with Qwen/Qwen3.5-397B-A17B

Pre-submission Checklist

  • My code follows the project style guidelines
  • I have performed a self-review of my code
  • I have added tests that prove the change is effective
  • New and existing tests pass locally
  • I have updated the documentation where needed
  • My changes generate no new warnings

Breaking Changes

  • No breaking changes

Additional Context

The committed results preserve the published baseline: 10 runs per case and model across 21 models via Nebius. Future runs can select models and run counts through environment variables.

@vetertann
vetertann marked this pull request as ready for review July 20, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TOON benchmark for generation tasks

1 participant