Repository navigation
fix(evals): Sweep eval tasks by age or run id instead of by prefix #1522
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
dbb5b87
fix(evals): Sweep eval tasks by age or run id instead of by prefix
jirispilka e5dd7d8
fix(evals): Keep the age-limit docstring true for tasks
jirispilka d0f45a8
fix(evals): Keep the skill reference and hint test in step with the s…
jirispilka 5d3573c
test(evals): Pin each sweep case once
jirispilka dbba464
Merge remote-tracking branch 'origin/master' into claude/epic-dijkstr…
claude ea7d974
test(evals): Match the tasks sweep test to the schedules one
claude ea9373f
refactor(evals): Consolidate fixture seeding and cleanup into a singl…
jirispilka a58bc88
docs(evals): Singular fixtures script wording
jirispilka File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| #!/usr/bin/env node | ||
| /* eslint-disable no-console */ | ||
| /** | ||
| * Fixtures for the `merge/tasks/*` and `merge/schedules/*` agent evals. | ||
| * | ||
| * Deletes the schedules and tasks a run left behind (delete rule in `sweep.ts`), then resets the | ||
| * two permanent read-only fixtures, creating them if missing. Schedules go before tasks and the | ||
| * task fixture before the schedule fixture, because the fixture schedule runs the fixture task. | ||
| * | ||
| * Usage: pnpm run evals:mcp-agent:fixtures [--run-id <id>] [--dry-run] | ||
| */ | ||
|
|
||
| import 'dotenv/config'; | ||
|
|
||
| import { ApifyClient, type ScheduleCreateOrUpdateData, ScheduleActions } from 'apify-client'; | ||
|
|
||
| import { findMissingEnvVars, sanitizeProcessEnv } from '../environment.js'; | ||
| import { parseRunIdArg } from '../run_id.js'; | ||
| import { FIXTURE_SCHEDULE_NAME, FIXTURE_TASK_NAME, isSweepable, type SweepCandidate } from './sweep.js'; | ||
|
|
||
| sanitizeProcessEnv(); | ||
|
|
||
| const FIXTURE_ACTOR = 'apify/normal-mode-test-actor'; | ||
| const FIXTURE_TASK_INPUT = { firstNumber: 1, secondNumber: 2 }; | ||
|
|
||
| /** Disabled on purpose: an enabled fixture would start a run on the eval account every night. */ | ||
| const FIXTURE_SCHEDULE = { | ||
| cronExpression: '0 3 * * *', | ||
| timezone: 'UTC' as const, | ||
| isEnabled: false, | ||
| isExclusive: true, | ||
| }; | ||
|
|
||
| /** `--dry-run` prints what the run would change and writes nothing. */ | ||
| const IS_DRY_RUN = process.argv.includes('--dry-run'); | ||
| /** Marks every line of a dry run, so its output cannot be read as changes that happened. */ | ||
| const DRY = IS_DRY_RUN ? '[dry run] ' : ''; | ||
|
|
||
| type Resource = SweepCandidate & { id: string }; | ||
|
|
||
| /** Deletes every sweepable item from `list` and returns the fixture if it is there. */ | ||
| async function sweep( | ||
| kind: string, | ||
| list: AsyncIterable<Resource>, | ||
| fixtureName: string, | ||
| runId: string | undefined, | ||
| remove: (id: string) => Promise<void>, | ||
| ): Promise<Resource | undefined> { | ||
| // Read every page before deleting anything: offset paging skips entries when the | ||
| // collection shrinks underneath it, and a missed leftover fails the next run. | ||
| const items = []; | ||
| for await (const item of list) items.push(item); | ||
|
|
||
| const now = Date.now(); | ||
| for (const item of items) { | ||
| if (!isSweepable(item, runId, now)) continue; | ||
| if (!IS_DRY_RUN) await remove(item.id); | ||
| console.log(`🗑️ ${DRY}Deleted leftover ${kind} "${item.name}" (${item.id})`); | ||
| } | ||
| return items.find((item) => item.name === fixtureName); | ||
| } | ||
|
|
||
| async function main() { | ||
| let runId: string | undefined; | ||
| try { | ||
| runId = parseRunIdArg(process.argv); | ||
| } catch (error) { | ||
| console.error(`❌ Error: ${error instanceof Error ? error.message : String(error)}`); | ||
| process.exit(1); | ||
| } | ||
| const missing = findMissingEnvVars(['APIFY_TOKEN']); | ||
| if (missing.length > 0) { | ||
| console.error(`❌ Error: missing environment variable(s): ${missing.join(', ')}`); | ||
| process.exit(1); | ||
| } | ||
| const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); | ||
|
|
||
| // The deletes below hit whatever account APIFY_TOKEN points at, so name it first. | ||
| console.log(`👤 ${DRY}Account: ${(await client.user('me').get()).username ?? 'unknown'}`); | ||
| if (runId) console.log(`🧹 ${DRY}Tearing down run "${runId}"`); | ||
|
|
||
| const schedule = await sweep('schedule', client.schedules().list(), FIXTURE_SCHEDULE_NAME, runId, async (id) => | ||
| client.schedule(id).delete(), | ||
| ); | ||
| let task = await sweep('task', client.tasks().list(), FIXTURE_TASK_NAME, runId, async (id) => | ||
| client.task(id).delete(), | ||
| ); | ||
|
|
||
| // Reset the input: an eval agent may have mutated the fixture (e.g. resolving a | ||
| // name collision by updating the existing task). | ||
| if (task) { | ||
| if (!IS_DRY_RUN) await client.task(task.id).update({ input: FIXTURE_TASK_INPUT }); | ||
| console.log(`♻️ ${DRY}Reset fixture task "${FIXTURE_TASK_NAME}" (${task.id})`); | ||
| } else { | ||
| if (!IS_DRY_RUN) { | ||
| const actor = await client.actor(FIXTURE_ACTOR).get(); | ||
| if (!actor) throw new Error(`Fixture Actor "${FIXTURE_ACTOR}" not found`); | ||
| task = await client.tasks().create({ | ||
| actId: actor.id, | ||
| name: FIXTURE_TASK_NAME, | ||
| title: 'Eval fixture (read-only)', | ||
| description: 'Permanent fixture for task-tool MCP agent evals. Do not modify or delete.', | ||
| input: FIXTURE_TASK_INPUT, | ||
| }); | ||
| } | ||
| console.log(`🌱 ${DRY}Created fixture task "${FIXTURE_TASK_NAME}"${task ? ` (${task.id})` : ''}`); | ||
| } | ||
|
|
||
| // Annotated, not inferred: an array literal widens `type` to the whole enum, which the | ||
| // client's discriminated action union rejects. | ||
| const actions: ScheduleCreateOrUpdateData['actions'] = task | ||
| ? [{ type: ScheduleActions.RunActorTask, actorTaskId: task.id }] | ||
| : []; | ||
| const scheduleFields = { | ||
| ...FIXTURE_SCHEDULE, | ||
| title: 'Eval fixture (read-only)', | ||
| description: 'Permanent fixture for schedule-tool MCP agent evals. Do not delete; do not modify.', | ||
| actions, | ||
| }; | ||
| // Reset everything an eval agent may have changed: enabled state, cadence, actions. | ||
| if (schedule) { | ||
| if (!IS_DRY_RUN) await client.schedule(schedule.id).update(scheduleFields); | ||
| console.log(`♻️ ${DRY}Reset fixture schedule "${FIXTURE_SCHEDULE_NAME}" (${schedule.id})`); | ||
| } else { | ||
| const created = IS_DRY_RUN | ||
| ? undefined | ||
| : await client.schedules().create({ ...scheduleFields, name: FIXTURE_SCHEDULE_NAME }); | ||
| console.log(`🌱 ${DRY}Created fixture schedule "${FIXTURE_SCHEDULE_NAME}"${created ? ` (${created.id})` : ''}`); | ||
| } | ||
|
|
||
| console.log(IS_DRY_RUN ? '✅ Dry run complete, nothing changed' : '✅ Fixtures ready'); | ||
| } | ||
|
|
||
| void main(); |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm glad to see this gone :)