[akzholai]

What this adds

DrexClassifier, an opt-in scorer in both Python and TypeScript that asks a Drex decision model whether an output meets a criterion. Drex answers typed questions (yes/no/unclear, multiple choice, or a 0-100 score) and returns probabilities, so the pass threshold can be chosen after the run, in Braintrust, instead of before it. Each score's metadata keeps the probabilities.

I work at Nace, which makes Drex. This is an independent scorer: it does not change any existing scorer, template or default.

from autoevals.drex import DrexClassifier, DrexQuestion

grounded = DrexClassifier(
    DrexQuestion.noul("Is every claim in the output supported by the context?"),
    name="Grounded",
)

How it works

• Plain HTTP to POST <base>/v1/systemone (httpx or urllib in Python, global fetch in TypeScript). No new runtime dependency.
• NACE_API_KEY is read from the environment; NACE_BASE_URL is optional for self-hosted Drex. The default model is drex-max.
• A Drex model can also be reached through OpenRouter; the key and the request URL follow the host actually being called, so a Drex key is never sent to another host.
• Examples are in examples/drex/, and the scorer is listed in SCORERS.md and README.md.

Other changes in this diff

js/version.ts is new so the TypeScript client can report autoevals/<version> the way the Python client already does through version.py. .github/scripts/check_version_sync.py, docs/PUBLISHING.md, AGENTS.md and the publish workflow's instruction text now say that package.json, py/autoevals/version.py and js/version.ts must match. Happy to split this out if you prefer.

Validation

• uv run --extra dev --extra scipy pytest py/autoevals/test_drex.py: 65 passed, 3 skipped (live tests, which need NACE_API_KEY).
• pnpm exec vitest run js/drex.test.ts: 56 passed, 3 skipped (live tests).
• python3 .github/scripts/check_version_sync.py: versions in sync (0.4.1).
• Live tests against Drex and against OpenRouter passed on the branch this was squashed from; they were not re-run on this rebased commit.
• tsc --noEmit reports one error in js/render-messages.test.ts, a file this PR does not touch.
• Not run: the full existing Python and JS suites, and pre-commit.

Agent review

• Authored with Claude Code (Claude Sonnet 5.5).
• The code went through several automated review rounds in a private fork before this PR; findings were fixed there. No separate review ran on this squashed commit after rebasing onto main.

🤖 Generated with Claude Code