Repository navigation
fix(evals): Sweep eval tasks by age or run id instead of by prefix - #1522
Open
jirispilka wants to merge 8 commits into
Open
jirispilka wants to merge 8 commits into
jirispilka wants to merge 8 commits into
Conversation
`tasks_fixtures.ts` deleted every non-fixture `eval-*` task on every run, so a merge-tier run starting while another was in flight destroyed its tasks; CI serialized the whole merge tier to hide it. Mirror the schedules sweep: a side-effect-free `tasks_sweep.ts` keeps young tasks in seed mode (6 h age), and `--run-id` deletes only that run's names. CI tears tasks down in an `always()` step with the run id, the local hint prints both teardown commands, and the `eval-merge-master` concurrency group goes. Claude-Session: https://claude.ai/code/session_01TJHJaSzUGq29JbjEwNPqUY
jirispilka
added a commit
that referenced
this pull request
Oct 12, 2026
## Why Closes #1426. `runAgentWithRetry` replays a trial's whole prompt after a transient agent error, but `{{uniq}}` was resolved once per trial, so the retry asked for the same schedule name. A create that succeeded before the failure collided with itself on the replay and failed the zero-tool-error gate. ## What changed `{{uniq}}` now resolves to `-<runId>-n<len>-t<trial>r<attempt>`, inside the retry loop, so each attempt names its own resource and the judge reads the name the attempt that ran used. Sibling: #1522 (#1424) adds `tests/unit/evals.tasks_sweep.test.ts` built with the two-argument `buildRunSuffix`; whichever lands second rebases that one call site. ## Notes for reviewers (human-written) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TJHJaSzUGq29JbjEwNPqUY
…a-hjc4n0-1424 # Conflicts: # evals/README.md
jirispilka
marked this pull request as ready for review
October 12, 2026 14:48
apify-service-account
added
the
tested
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Closes #1424.
tasks_fixtures.tsdeleted every non-fixtureeval-*task on every invocation, so a merge-tier run that started while another was in flight destroyed the first run's tasks. CI hid it by serializing the merge tier (eval-merge-master), which also queued back-to-back master pushes. #1423 fixed this for schedules; tasks still swept by prefix.What changed
Tasks now sweep by age or
--run-id, like schedules: without--run-id, onlyeval-*tasks older than 6 h; with it, only that run's names at any age. The permanent fixture is exempt by name. Theeval-merge-masterconcurrency group is gone.The eight
merge/tasks/*items that create a task name it…-{{uniq}}in Langfuse (edited 2026-10-11), so each run and trial works on its own tasks.Why one script instead of mirroring #1423
The issue asked to mirror
schedules_fixtures.ts. Doing that left two scripts that were the same code apart from the fixture name, with a hidden ordering contract (tasks before schedules on seed, schedules before tasks on teardown) repeated in four workflow steps, the README and a docstring. So they're merged:evals/scripts/fixtures.ts(pnpm run evals:mcp-agent:fixtures) sweeps schedules, then tasks, then resets both fixtures. One order, correct for seed and teardown.evals/scripts/sweep.tsholds the one delete rule for both kinds.always()teardown step.Trade-off: a failing schedule sweep now skips the task sweep. The next seed deletes anything older than 6 h, so the cost is delayed cleanup, not leaks.
Proof it works
--run-id shep1424amatched exactly this run's eight tasks and the real teardown deleted only those, fixture reset. 19/20 items passed (the miss was agent behavior). https://langfuse.apify.dev/project/cmshkde21000krg07shb46d8g/datasets/cmtt5bb54020itx07em540k7g/runs/3a42cc716f2905afAI disclosure: written with Claude Code (shepherd pipeline, four reviews); the Langfuse edits and the live proof were run by the agent.
🤖 Generated with Claude Code