Skip to content

LLM benchmark: new evals + improved tooling#5583

Open
bradleyshep wants to merge 76 commits into
masterfrom
bradley/new-llm-benchmark-evals-0717
Open

LLM benchmark: new evals + improved tooling#5583
bradleyshep wants to merge 76 commits into
masterfrom
bradley/new-llm-benchmark-evals-0717

Conversation

@bradleyshep

Copy link
Copy Markdown
Contributor

Description of Changes

Adds 37 LLM benchmark evals (t_047t_083) covering SpacetimeDB functionality across:

  • Table design, normalization, indexes, defaults, binary data, and special types
  • Atomic and nested reducer behavior, cascade and batch deletion, deterministic context, and result tables
  • Procedural views, joins, filtering, timestamp windows, aggregates, and view primary keys
  • Secure projections, row-level security, lifecycle handlers, scheduled work, and connection-scoped presence
  • Procedures, HTTP requests, handlers, routers, webhooks, and external upload flows
  • Automatic, incremental, and hot-swap-compatible migrations

Each eval includes consistent prompts, language-specific goldens, schema validation, and behavioral scoring for TypeScript, Rust, and C#.

This PR also improves the benchmark runner by isolating concurrent builds and databases, enforcing timeouts, handling transient build failures more reliably, and running work in parallel. The periodic workflow now runs each language in a separate job.

The server skills were updated with general, non-prescriptive guidance for the functionality covered by these evals.

API and ABI breaking changes

None.

Expected complexity level and risk

Complexity: 2/5.

This is a secondary testing tool and is isolated from the product. These changes do not affect SpacetimeDB runtime behavior or public APIs. The main risk is incorrect eval scoring or benchmark workflow failures.

Testing

  • All 43 benchmark unit tests pass
  • Formatting and Clippy checks pass
  • Updated goldens build, publish, and pass their scorers in TypeScript, Rust, and C#
  • GPT-5.5 task pass rates: TypeScript 100%, Rust 100%, C# 97.3%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant