Skip to content

Pin workflow steps to the Agent version they were saved with - #1674

Open
tataru-dan wants to merge 2 commits into
mainfrom
feat/1673-agent-versioning
Open

tataru-dan wants to merge 2 commits into
mainfrom
feat/1673-agent-versioning

Conversation

@tataru-dan

Copy link
Copy Markdown
Collaborator

Closes #1673

Description

Agent definitions have been versioned since ADR-0029 (every save or eval hashes the config and appends an immutable AgentVersion row), but that versioning only fed Eval Runs. A production Workflow step resolved its Agent live at execution time and never recorded which version actually ran, so editing an Agent after a Workflow used it could silently change what a step does mid-run, and retrying a failed step could rerun it on a different Agent than the one that originally ran it.

This PR closes that gap end to end, in two commits.

Commit 1: track which Agent version a workflow step actually ran

  • Every Agent save (or eval) that changes its config appends a new AgentVersion (config snapshot + hash) to its history.
  • AgentRun now stamps the agentId and agentVersion the step actually ran with, recorded by the agent runner and exposed through the CLI (agent-run-get, agent-run-list).
  • The Agent's "Versions" tab shows every Workflow step using it ("Used by"), flags one that's behind the Agent's current version, and lets you diff any two versions (prompt, model, skills) side by side.
  • A running Workflow's step status shows v{N} for the version that ran, flagged amber when the Agent has since moved on.
  • Retrying a failed step now refuses to silently rerun it on a different Agent version than the one that ran it before, unless allowAgentDrift is explicitly passed.
  • The Agent Assistant can now propose updating an existing Agent's prompt or model (propose_agent_update), not just creating new ones or adding a Skill — this is the "unblock Agent editing from the assistant" piece of the epic, now that a change is reproducible.

Commit 2: pin a workflow step to the Agent version it was saved with

  • A step now carries its own agentVersion, written automatically when its Agent is picked in the step editor's "Agent" dropdown, and changed only by a manual edit via a new "Agent version" picker next to it (or an explicit "Track current" choice for a step that should keep following the Agent live).
  • Execution resolves the Agent's identity prompt, model, output schema, MCP servers and Skills from that pinned version's recorded snapshot instead of its live document — so editing an Agent, even multiple versions ahead, never changes what an already-saved step runs. Verified end to end: pinned a step to v4, edited the Agent to v6, reran the step, and confirmed via the run's debug output that it still executed with the v4 prompt.
  • The retry drift guard now compares against the step's own pin (not the Agent's current version), so a pinned step never falsely trips it.
  • Fixed two bugs found while building this: the version badge on a workflow-editor step overlapped its hover buttons (trash/move), and the Agent Versions tab's "Used by" row linked to a dead route and showed no workflow name — it now links straight to the run and step that used it.

How to test

  1. Open a workflow with an agent step, pick an Agent in the step editor — note the pinned version shown next to it.
  2. Edit that Agent (change its prompt) and save — it gets a new version.
  3. Reopen the workflow step: it still shows the old pinned version, not the new one.
  4. Run the workflow and check the step's debug output — it used the old (pinned) prompt.
  5. On the Agent's "Versions" tab, confirm "Used by" links to the actual run/step and flags it as behind.
  6. Retry a failed step on that workflow without allowAgentDrift — it should succeed, using the pinned version, not refuse.

Definition of Done

  • Typechecked clean across platform-core, agent-runtime, platform-api, platform-ui, cli.
  • Unit and integration tests pass for every package touched, including new tests for the pinning behaviour, the retry drift guard, and the Agent Versions tab fixes.
  • Verified manually end to end against the local dev app (Playwright): pin persists across save/reload, execution genuinely uses the pinned prompt after the Agent changed, retry respects the pin, "Used by" link navigates correctly.
  • No regressions for unpinned (pre-existing) steps: they keep today's live-tracking behaviour until someone picks their Agent again.
  • Docs: no existing living doc made a claim this change contradicts; a short note on step-level Agent version pinning could be added to the evaluation reference docs as a follow-up, not blocking.

Closes

Closes #1673 — all workstreams (A1, B1, B2, C1, C2, C3, D1, E1) are covered by this PR.

Each Agent save now records a version (config + hash) in its history.
AgentRun stamps the Agent id and version it ran with, surfaced through
the CLI, the Agent's Versions tab (used by, diff between versions),
and step status with a staleness flag when the Agent moved on.
Retrying a step now refuses to silently rerun it on a different Agent
version unless allowAgentDrift is explicitly passed.
…ed with

A step now carries its own agentVersion, set when its Agent is picked
and changed only by a manual edit in the step editor's version picker.
Execution resolves identity, model, MCP servers and Skills from that
pinned version's snapshot instead of the Agent's live document, so a
later Agent edit never changes what an already saved step runs. Retry
compares against the pin instead of the Agent's current version, so a
pinned step never trips the drift guard. Also fixes two display bugs
found while building this: hover buttons on a workflow step overlapping
its version badge, and the Agent Versions tab linking to a dead route
with no workflow name shown.
@tataru-dan tataru-dan self-assigned this Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Epic: Agent Versioning, reproducible Workflow runs and unblocking Agent Assistant edits

1 participant