Skip to content

fix(server): tool calls a released session cuts off no longer keep the thread Waiting - #29036

Open
NotReliable wants to merge 8 commits into
pingdotgg:mainfrom
NotReliable:fix/close-orphaned-turn-items-on-handoff
Open

NotReliable wants to merge 8 commits into
pingdotgg:mainfrom
NotReliable:fix/close-orphaned-turn-items-on-handoff

Conversation

@NotReliable

Copy link
Copy Markdown

Problem

After t3_worktree_handoff, the thread shows "Waiting on background task mcp__t3-code__t3_worktree_handoff" for good, also after later turns complete (#15993). The handoff's dynamic_tool item stays running with completedAt: null.

The handoff detaches the Claude session. releaseEntry fails the run's subscribers before the adapter can report how its open tool calls ended. Run finalization keeps command_execution and dynamic_tool items open for a late result, so nothing ever ends them. pendingBackgroundTurnItems then lists the item, and the thread waits. The same leftover occurs on any release of such a session while a tool call runs, for example a provider switch.

Seen on 0.1.3-nightly.20261026.3717 (Linux, claudeAgent): in one state database, 6 of 6 handoff tool items were running inside a failed run.

Change

The release path already clears the dead session's background-task roster ("Dead processes cannot report task endings"). It now also interrupts the session's cut-off tool calls, with the same gate.

 releaseEntry(session)
   readReleasedProviderThreads        # gate: backgroundTasksEndWithProcess, not server_shutdown
   failSubscribers
   closeScope                         # waits for the process to stop
   writeReleaseRecords
     writeReleasedRosterEvents        # existing: clear the task roster
+    writeReleasedToolCallEvents      # new: open command_execution / dynamic_tool items
+      repeat up to 3 passes
+        read open tool calls on the released provider threads with startedAt <= releasedAt
+        stop when none is open
+        for each: writeIfTurnItemUnchanged(item, status: "interrupted")
  • Which items: open command_execution and dynamic_tool items on the session's provider threads that started at or before the release. Items that a replacement session starts later stay open. Subagent rows are not changed, because run finalization already ends run-owned ones.
  • Status: interrupted, the status Stop already gives items on a dead session.
  • Late results win: EventSink.writeIfTurnItemUnchanged (same pattern as writeIfProviderThreadUnchanged, now one shared helper) commits only if the stored item is unchanged. A completion from the dead session's queued events rejects only the interrupt of its own item. Each pass ends with a new read, so items that queued events store during cleanup are also closed.
  • Which runtimes: only runtimes that set backgroundTasksEndWithProcess (Claude and Muse; Muse's host runs all its session work). Runtimes whose work can outlive the session, and server_shutdown, keep their items as before. The flag's comment now says that it covers tool calls too.

Door: two-way. No schema, contract, or setting change; the only new API is one internal EventSink method.
Revert: revert the PR. Items that it already marked interrupted stay interrupted, the same state that Stop writes today.
Blast radius: server-only. Every release of a Claude or Muse session (detach, switch, idle release, close) now does one or two more filtered turn-item reads per released provider thread.

Scope and approval

Fixes #15993 (triaged bug). The maintainer triage there names this direction: publish the tool cancellations on an exclusive-session release.

This addresses the objection that closed #17306 ("marks tool calls interrupted after ingestion errors without stopping the provider"). The change does not touch the ingestion-failure path. It runs only in releaseEntry, after the session scope close has waited for the provider process to stop, so the provider loss is confirmed. The integration harness follows the one in #17306.

Out of scope, by choice:

Verification

Red first. The new assertions fail on main and pass with the change:

Test main This PR
testkit/ProviderSwitch.integration › interrupts the handoff tool call that releases its own session (real thread.metadata.update → detach, asserts derivePendingBackgroundWork is []) expected 'running' to equal 'interrupted' pass
ProviderSessionManager › clears the background work of a session a detach releases (cut-off → interrupted; completed, other-session and replacement items unchanged) fail pass
ProviderSessionManager › keeps provider updates that land while release clears (4 late completions win, a late output update and items stored during cleanup are still interrupted, the 3-pass bound holds) fail pass
ProviderSessionManager › keeps the background work after the release of a runtime whose tasks can outlive it / server shutdown pass pass
  • I extended the existing roster tests and their fixture, and added no parallel cases.
  • Hand mutants on the new logic are all killed: no started-before-release filter, no provider-thread filter, no session filter, unconditional write, one write for all items, stop when a pass has no conflict, retry only the earlier ids, one, two or four passes, no pass bound, cancelled status, no server_shutdown gate.
  • The 4 tests passed 20 of 20 repeated runs.
  • vp run check:changed: format, lint, knip, graphql, typecheck (25 packages) and the tests beside the changed files all pass. 9 affected test files (ProviderSessionManager, ProviderSwitch.integration, BackgroundWorkStop.integration, SelectionRestart.integration, RunExecutionService, ProviderRuntimeRecoveryService, WorktreeMcpService, FoundationPersistence, ProviderEventIngestor): 275 tests pass.

Not checked:

  • A manual handoff in the desktop or mobile app.
  • An item that the dead session stores only after the last read of the third pass stays open, and cleanup then logs provider-session-release-tool-call-conflict with the count.

Skills: core-v2.0.0

Model and harness: Claude Opus 5.5 through Claude Code in T3 Code.

🤖 Generated with Claude Code

Spike and others added 6 commits November 2, 2026 03:49
…e thread Waiting

When T3 releases a provider session whose background work ends with its
process, such as the detach after t3_worktree_handoff or a provider switch,
the session cannot report how its open tool calls ended. Run finalization
keeps command and dynamic tool items open for a late result, so these items
stayed running. The thread then showed "Waiting on background task" until a
restart.

The release path already clears the dead session's background-task roster.
It now also marks that session's open command_execution and dynamic_tool
items as interrupted, as Stop does for items on a dead session. It reads the
item ids when the session leaves the live map, so items of a replacement
session stay open. Runtimes whose tasks can outlive them, and server
shutdown, keep their items as before.

Fixes pingdotgg#15993

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repairs:
- Select the cut-off items at write time: open tool calls on the released
  provider threads that started at or before the release. This includes an
  item that the dead session's queued events stored after the release began.
  Items that a replacement session started later stay open.
- Write the interrupt only if each item row is unchanged
  (EventSink.writeIfTurnItemsUnchanged), with three tries like the roster.
  A late completion from the dead session now wins over the interrupt.
- Use Effect.fn spans for both release writers, and set driver on the events.
- State in the backgroundTasksEndWithProcess comment that the flag also
  covers tool calls.
- Extend the existing roster tests instead of adding new ones, add the
  server shutdown row, and keep one integration case (the handoff).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l write

Review repair: one conditional write for all cut-off items let three late
completions use up the three tries, and the other open items stayed
running. Each item now gets its own conditional write. A late completion
rejects only the interrupt of its own item, and that item is then no longer
open. Items that are still open after a conflict, such as after a late
output update, are read again and tried again, up to three times.

EventSink.writeIfTurnItemUnchanged replaces the batch write, and both
conditional writes share one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repair: the retry read only the items that conflicted. A conflict
shows that the dead session's queued events are still landing, so they can
also store more cut-off items. The retry now reads all open cut-off items
again. Items that the earlier pass interrupted are no longer open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review repair: a pass with no conflict returned at once, so an item that the
dead session's queued events stored during that pass was never read. Each
pass now ends with a new read, and cleanup stops only when that read finds
no open cut-off item, or after three passes. The warning then names the
count of open tool calls. The raced test checks statuses by item id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review nit: the raced test stores one more cut-off call during the third
pass and checks that it stays open, so the pass bound is tested. The comment
on the late calls now says that only the first four writes complete one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Nov 2, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 6b0412d

Macroscope's review found this PR approvable — Released sessions now mark their cut-off tool calls interrupted instead of leaving threads Waiting. Cleanup is adapter-gated and uses transactional conditional writes to protect concurrent updates.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Walkthrough

Provider session release now captures matching provider threads, clears eligible task rosters, and interrupts eligible active tool calls through conditional turn-item writes. Tests cover release cases, subscription handling, cleanup races, and a provider handoff.

Changes: Provider session release cleanup

Layer / File(s) Summary
Conditional turn-item writes
apps/server/src/orchestration-v2/EventSink.ts, apps/server/src/orchestration-v2/ProjectionStore.ts
The event sink adds a conditional turn-item write operation and shares its payload-check and event-commit path with provider-thread writes. encodeTurnItemPayload is exported.
Release-time roster and tool-call cleanup
apps/server/src/orchestration-v2/ProviderSessionManager.ts, packages/provider-core/src/server/ProviderAdapter.ts
Release processing captures matching provider threads, removes captured task IDs, and interrupts eligible active tool calls. Subscriber termination runs cleanup after queues drain or close. The adapter documentation describes the process-lifetime condition and release behavior.
Release behavior and race coverage
apps/server/src/orchestration-v2/ProviderSessionManager.test.ts, apps/server/src/orchestration-v2/testkit/ProviderSwitch.integration.test.ts
Tests cover calls preserved or interrupted during release, late completions across cleanup passes, and a handoff that releases its own session.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Low

Sequence Diagram(s)

sequenceDiagram
  participant ProviderSessionManager
  participant EventSinkV2
  participant ProjectionStore
  ProviderSessionManager->>EventSinkV2: Request conditional turn-item write
  EventSinkV2->>ProjectionStore: Encode expected turn item
  ProjectionStore-->>EventSinkV2: Return encoded payload
  EventSinkV2->>EventSinkV2: Compare payload and append events if unchanged
  EventSinkV2-->>ProviderSessionManager: Return commit result
Loading

Suggested labels: bugfix

Suggested reviewers: juliusmarminge, maria-rcks

Merge Risk: Minimal · up to b19b6

No issue requiring a change before merge was identified. The release cleanup’s three-pass limit remains a documented boundary.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title names the fix and its scope: tool calls left open when a session is released no longer keep the thread in Waiting. It matches the changeset, which interrupts open command_execution and dynam…
Description check Passed The description follows the repository template closely. It has Problem, Change, Scope and approval, and Verification sections. The Problem section gives the symptom, the cause, and a reproduction env…
Linked Issues check Passed The active direct requirement is #15993. ProviderSessionManager now captures provider threads at release and interrupts eligible open command_execution and dynamic_tool items. The cutoff exclude…
Out of Scope Changes check Passed The production changes support #15993 release cleanup and its queued-event race handling. The EventSink helper and exported payload encoder support conditional turn-item writes. The test changes cov…
Full details: Title check

Explanation

The title names the fix and its scope: tool calls left open when a session is released no longer keep the thread in Waiting. It matches the changeset, which interrupts open command_execution and dynamic_tool items during release for runtimes whose background work ends with the process. The title follows the conventional commit format the template asks for.

Full details: Description check

Explanation

The description follows the repository template closely. It has Problem, Change, Scope and approval, and Verification sections. The Problem section gives the symptom, the cause, and a reproduction environment. The Change section explains the release-path flow and the conditional write. Scope cites the triaged bug #15993 and the maintainer direction, and it explains how the change differs from closed #17306. Verification gives red-then-green results for named tests, mutation checks, repeated runs, and the check command, and it lists what was not checked. The template's optional agent-attribution line is present. The description is detailed and mostly complete, so it passes.

Full details: Linked Issues check

Explanation

The active direct requirement is #15993. ProviderSessionManager now captures provider threads at release and interrupts eligible open command_execution and dynamic_tool items. The cutoff excludes items started after release and preserves replacement-session items. The three-pass read and writeIfTurnItemUnchanged check account for queued events and allow late completions to win. The integration test covers a handoff release and asserts interrupted with no pending background work. #17306 is closed and supplies historical context only.

Full details: Out of Scope Changes check

Explanation

The production changes support #15993 release cleanup and its queued-event race handling. The EventSink helper and exported payload encoder support conditional turn-item writes. The test changes cover release, handoff, cutoff, late-result, replacement-session, runtime, and server-shutdown behavior. The adapter comment documents the existing runtime gate. No unrelated schema, client, ingestion-failure, continuation, or out-of-scope runtime behavior change is shown.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

ast-grep (0.51.1)

ast-grep could not parse rule config: /ast-grep-rules/ast-grep-essentials/rules/csharp/security/jwt-tokenvalidationparameters-no-expiry-validation-csharp.yml

CodeRabbit Verify

  • Run CodeRabbit Verify to test the changes in this pull request.
Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 6b0412d and b19b6d3.

Files selected for processing: 5
  • apps/server/src/orchestration-v2/ProjectionStore.ts
  • apps/server/src/orchestration-v2/ProviderSessionManager.test.ts
  • apps/server/src/orchestration-v2/ProviderSessionManager.ts
  • apps/server/src/orchestration-v2/testkit/ProviderSwitch.integration.test.ts
  • packages/provider-core/src/server/ProviderAdapter.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

@maria-rcks

maria-rcks commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

merged current main without rebasing, preserving both sides of the test-adapter conflict. fixed a release race where queued provider events could reopen a tool call after cleanup. released subscriptions now clean up once on close, including drain, runtime failure, graceful provider stop, and early consumer exit. late completions and replacement-session work keep their existing protections.

blacksmith verification on b19b6d3759: vp run check:changed github/main (format, lint, knip, graphql, affected typechecks, and 151 tests). the queued-update regression failed before the fix and passed afterward. two independent code reviews found no remaining issues. real-provider/client evidence below is from the previous verification pass; it was not rerun for this follow-up.

Verified in a real dev environment

verified unchanged head 6b0412db8d2d9a1b6f13f2bc9a07c9cfa115c4d5 against github/main at cbd3e149a1 on linux. used separate checkouts, vp i, one background dev server at a time, repository playwright with system chrome, and projects/threads created through the web app against a real git repository. no mocks or seeded records. real codex gpt-6.1-sol and claude sonnet 4.6 turns used the local proxy.

  • reproduced on main: while claude had a foreground shell command open, sent the public thread.metadata.update websocket command used by worktree handoff, targeting a worktree created by the actual handoff tool. the run failed on detach, but the command stayed running with completedAt: null, including after a successful recovery turn.
  • on this head, the matching command became interrupted with a completion time. recovery read the worktree directory and readme and completed, with no stale running row.
  • also detached claude through the public websocket API during a real t3_thread_wait call. that dynamic tool became interrupted. a separate codex command stayed running during cleanup, then completed normally with its expected output.
  • normal t3_worktree_status and t3_worktree_handoff worked, and the completed handoff result was preserved. clicking stop interrupted a new foreground command.
  • blacksmith: vp test run apps/server/src/orchestration-v2/ProviderSessionManager.test.ts apps/server/src/orchestration-v2/testkit/ProviderSwitch.integration.test.ts: 92 passed, 2 files passed. no code changes or pushes needed.

main after recovery, with the orphan still running:

main: recovered turn above the stale running command

pr after the same release and recovery, with no stale running command:

pr: recovered turn and terminal command item

main and pr release and recovery recording; pr segment at 4x

short runtime transcript

Not verified

the exact race where the handoff tool itself loses its result: ordinary real handoffs on both revisions delivered that result before detach. the command cutoff and dynamic-tool cutoff paths were exercised separately through the running server. the handoff run still ends with a provider-stream failure, as explicitly excluded from this pr. muse, native desktop/mobile clients, and relay/tunnel modes were not exercised. cleanup contention and the three-pass bound were covered by existing tests, not induced in the live app.

@michael0903

Copy link
Copy Markdown

Still reproduces on 0.1.8-nightly.20261106.4262 (macOS, Claude provider). After a t3_worktree_handoff, the thread footer kept showing "Waiting · mcp__t3-code__t3_worktree_handoff" for more than 30 minutes after the agent had finished its reply, so the thread stayed in the sidebar's Working section.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Stale "Waiting on background task T3 Worktree Handoff" banner persists after a Claude worktree handoff

3 participants