Repository navigation
fix(server): auto-settle no longer cuts off an agent whose awaited background task just finished - #29820
fix(server): auto-settle no longer cuts off an agent whose awaited background task just finished#29820dominic-r wants to merge 6 commits into
Conversation
…uto-settle Providers drop finished background work from the provider thread's roster shortly before the turn that reports its result is requested (about 100ms for Claude). Auto-settle reacted to the empty roster inside that gap, detached the session, and the agent never received the result. Remember when holding work leaves a provider thread's roster and keep the thread out of auto-settlement until a turn is requested after that moment, for at most the queued-turn grace.
ApprovabilityVerdict: Approved at Macroscope's review found this PR approvable — Automatic settlement briefly waits for the reporting turn after awaited background work finishes, avoiding premature session detachment. The per-thread delay is bounded by the existing two-minute grace window and leaves settlement settings unchanged. You can add or adjust custom eligibility rules. Learn more. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the
Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
…r-background-report
The hold was recorded in memory from the domain event stream, but full and merge sweeps read the projection, which can already show the empty roster while the event consumer lags. The provider thread projection now records when its roster stops holding work, in the same transaction as the roster update, and every sweep reads that time from the settlement candidates. The in-memory tracking is gone.
|
@coderabbitai review |
Action not completed
|
Action not completed
|
Action not completed
|
… is queued A report requested behind another run stays queued until that run ends, so its request time alone does not mean the report has started. A full or merge sweep in that window could still settle the thread.
|
@macroscope-app please re-run approvability and correctness on 2c5988e. |
|
Sorry, I'm unable to act on this request because you do not have permissions within this repository. |
Action not completed
|
|
Note Written by merged current main without rebasing in verification on blacksmith: live provider/client reproduction remains unverified; validation used the existing server regression tests. ci is running on the pushed head. |
…r-background-report
Problem
Auto-settle can settle a thread at the moment its awaited background work finishes, before the agent hears the result. Settling detaches the provider session, so the provider process exits and the turn that would have reported the result never starts. The agent stops mid-task without any notice. A server restart at least sends a note about cancelled background work; a settle sends nothing.
Reproduction: with "settle when the PR merges" on (or an inactivity window that has passed), an agent starts a background command, marks it with
t3_background_wait, and ends its turn. The PR merges, or the thread is already past the inactivity window. When the command finishes, the provider removes it from the provider thread's roster, emittingprovider-thread.updatedwith an emptypendingBackgroundTasks. That event enqueues a settlement sweep. The thread now has no holding work and no active run, so the sweep dispatchesthread.auto-settle, about 15–50 ms after the roster emptied. Theprovider-continuation:message that would wake the agent normally arrives about 100 ms after the roster empties (Claude), but by then the session is detached.In one install's event log over a week, 72 auto-settles fired inside this gap, every time with awaited work. Several were delegated child threads, so their parents never got a result either. When nothing settles in between, the wake turn arrives a median of 0.13 s and at most 11 s after the roster empties.
Change
The provider thread projection now records when a provider thread's roster stops holding work, meaning
backgroundWorkHoldsCompletiongoes from true to false. It does this in the same transaction as the roster update. A newholds_completioncolumn keeps the previous state, so the upsert can spot the transition, andheld_work_ended_atstores its time (migration 79).Settlement candidates carry the latest of those times. Every sweep, whether incremental, full or after a merge, leaves the thread out of auto-settlement until a turn is requested after that moment (a user-role message or a later run), and for as long as that run is still queued behind another one. Because the time comes from the same committed state as the empty roster, a sweep that sees one also sees the other, however far behind the settlement event consumer is.
The hold reuses the existing two-minute queued-turn grace. If no report turn comes, for example because the run was stopped, the thread settles as before on a later sweep. Nothing changes in what
pendingBackgroundTasksorbackgroundWorkHoldsCompletionreport, and manual settle and archive still detach immediately.Known limit: a roster that empties for some other reason (a stopped run, a new turn starting) delays settlement by up to two minutes.
Scope and approval
Fixes #26083, which reports this race and hasn't been triaged yet, so I'm also submitting this under the small, focused fix exception. Fixes #29899, the full- and merge-sweep gap left by an earlier version of this PR.
docs/user/thread-sidebar.mdalready says background work the agent is waiting on prevents automatic settlement. This restores that behavior for the moment the work finishes. The change is the settlement service, the provider-thread projection and a migration that adds two columns, with no contract, client, or settings changes.Verification
I did not run a live provider turn against a patched server, so the end-to-end path is shown by the event-log timings above, not observed after the fix. I also did not check whether other adapters have a similar gap on paths that do not go through the provider-thread roster, such as delegated-task completions.
Written by Claude Opus 5.5 in Claude Code, run from T3 Code.