Skip to content

fix(server): auto-settle no longer cuts off an agent whose awaited background task just finished - #29820

Open
dominic-r wants to merge 6 commits into
pingdotgg:mainfrom
dominic-r:fix/settle-waits-for-background-report
Open

dominic-r wants to merge 6 commits into
pingdotgg:mainfrom
dominic-r:fix/settle-waits-for-background-report

Conversation

@dominic-r

@dominic-r dominic-r commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Auto-settle can settle a thread at the moment its awaited background work finishes, before the agent hears the result. Settling detaches the provider session, so the provider process exits and the turn that would have reported the result never starts. The agent stops mid-task without any notice. A server restart at least sends a note about cancelled background work; a settle sends nothing.

Reproduction: with "settle when the PR merges" on (or an inactivity window that has passed), an agent starts a background command, marks it with t3_background_wait, and ends its turn. The PR merges, or the thread is already past the inactivity window. When the command finishes, the provider removes it from the provider thread's roster, emitting provider-thread.updated with an empty pendingBackgroundTasks. That event enqueues a settlement sweep. The thread now has no holding work and no active run, so the sweep dispatches thread.auto-settle, about 15–50 ms after the roster emptied. The provider-continuation: message that would wake the agent normally arrives about 100 ms after the roster empties (Claude), but by then the session is detached.

In one install's event log over a week, 72 auto-settles fired inside this gap, every time with awaited work. Several were delegated child threads, so their parents never got a result either. When nothing settles in between, the wake turn arrives a median of 0.13 s and at most 11 s after the roster empties.

Change

The provider thread projection now records when a provider thread's roster stops holding work, meaning backgroundWorkHoldsCompletion goes from true to false. It does this in the same transaction as the roster update. A new holds_completion column keeps the previous state, so the upsert can spot the transition, and held_work_ended_at stores its time (migration 79).

Settlement candidates carry the latest of those times. Every sweep, whether incremental, full or after a merge, leaves the thread out of auto-settlement until a turn is requested after that moment (a user-role message or a later run), and for as long as that run is still queued behind another one. Because the time comes from the same committed state as the empty roster, a sweep that sees one also sees the other, however far behind the settlement event consumer is.

The hold reuses the existing two-minute queued-turn grace. If no report turn comes, for example because the run was stopped, the thread settles as before on a later sweep. Nothing changes in what pendingBackgroundTasks or backgroundWorkHoldsCompletion report, and manual settle and archive still detach immediately.

Known limit: a roster that empties for some other reason (a stopped run, a new turn starting) delays settlement by up to two minutes.

Scope and approval

Fixes #26083, which reports this race and hasn't been triaged yet, so I'm also submitting this under the small, focused fix exception. Fixes #29899, the full- and merge-sweep gap left by an earlier version of this PR. docs/user/thread-sidebar.md already says background work the agent is waiting on prevents automatic settlement. This restores that behavior for the moment the work finishes. The change is the settlement service, the provider-thread projection and a migration that adds two columns, with no contract, client, or settings changes.

Verification

  • Projection: a new test applies provider-thread updates to a real projection store. Ending an awaited command records its end time. Ending a command nobody waited on records nothing, and a later update with nothing held leaves the recorded time alone.
  • Settlement worker: both tests read the hold from settlement candidates, with no roster event delivered, which is the lagging-consumer case:
    • A thread whose PR merged and whose awaited command just ended survives the startup full sweep, and then a merge sweep while its report turn is queued. It settles at the next full sweep once that report turn has run.
    • An inactive thread settles once the grace passes with no report turn.
  • Without the fix: I ran these tests with the hold check removed from the sweep and the end-time recording removed from the upsert, and all three failed. Removing only the queued-run check fails the merge-sweep test. With the fix they pass, along with the rest of the settlement, projection-store and migration tests.
  • Checks: format, lint, knip and the server typecheck pass.

I did not run a live provider turn against a patched server, so the end-to-end path is shown by the event-log timings above, not observed after the fix. I also did not check whether other adapters have a similar gap on paths that do not go through the provider-thread roster, such as delegated-task completions.

Written by Claude Opus 5.5 in Claude Code, run from T3 Code.

…uto-settle

Providers drop finished background work from the provider thread's
roster shortly before the turn that reports its result is requested
(about 100ms for Claude). Auto-settle reacted to the empty roster
inside that gap, detached the session, and the agent never received
the result. Remember when holding work leaves a provider thread's
roster and keep the thread out of auto-settlement until a turn is
requested after that moment, for at most the queued-turn grace.
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Nov 3, 2026
@macroscopeapp

macroscopeapp Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 04c89c3

Macroscope's review found this PR approvable — Automatic settlement briefly waits for the reporting turn after awaited background work finishes, avoiding premature session detachment. The per-thread delay is bounded by the existing two-minute grace window and leaves settlement settings unchanged.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

Provider-thread projections now record when held work ends. Settlement candidates include that time, and settlement sweeps defer eligible threads while report activity is pending or the grace period remains active.

Changes: Awaited-work settlement

Layer / File(s) Summary
Record held-work completion
apps/server/src/persistence/Migrations/080_ProviderThreadHeldWorkEndedAt.ts, apps/server/src/persistence/Migrations.ts, apps/server/src/persistence/reconcileV2PreviewMigration.test.ts, apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/ProjectionSettlement.test.ts
Migration 80 adds holds_completion and held_work_ended_at. Provider-thread updates store the hold state and record the event time when a held state ends. Settlement candidates include the latest recorded end time. Tests cover timestamp recording and migration registration.
Defer settlement pending report activity
apps/server/src/orchestration-v2/ThreadSettlementService.ts, apps/server/src/orchestration-v2/ThreadSettlementService.test.ts
The sweep excludes candidates awaiting a report turn during the grace period. Tests cover queued report activity and settlement after the grace period.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant BackgroundTask
  participant ProjectionStore
  participant ThreadSettlementService
  BackgroundTask->>ProjectionStore: Provider-thread roster stops holding completion
  ProjectionStore->>ProjectionStore: Record held_work_ended_at
  ProjectionStore->>ThreadSettlementService: Provide settlement candidate with held-work end time
  ThreadSettlementService->>ThreadSettlementService: Defer settlement while awaiting report activity
Loading

Possibly related PRs

Suggested labels: bugfix

Suggested reviewers: t3dotgg, yash-singh1

Merge Risk: Minimal · up to a132b

The reviewed settlement race is addressed, and no additional merge-blocking defect was established.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title names the main change: auto-settle no longer cuts off an agent right after its awaited background task finishes. It is a conventional commit title in plain language and matches the changeset…
Description check Passed The description follows the template sections (Problem, Change, Scope and approval, Verification). It explains the race, the projection and migration change, and the linked issues. It gives concrete v…
Linked Issues check Passed The changes meet the coding requirements for [#26083, #29899]. ProjectionStore records held_work_ended_at when awaited work stops holding completion. Settlement candidates carry the latest timesta…
Out of Scope Changes check Passed The changes stay within [#26083, #29899]. The projection updates, settlement logic, migration 80, migration registration, migration expectations, and tests implement or verify the requested settlement…
Full details: Title check

Explanation

The title names the main change: auto-settle no longer cuts off an agent right after its awaited background task finishes. It is a conventional commit title in plain language and matches the changeset, which records the held-work end time and defers settlement until a report turn is requested.

Full details: Description check

Explanation

The description follows the template sections (Problem, Change, Scope and approval, Verification). It explains the race, the projection and migration change, and the linked issues. It gives concrete verification: the three regression tests failed without the fix, and format, lint, knip and typecheck pass. It also states the known limits: a roster that empties for other reasons delays settlement by up to two minutes, and no live provider turn was run. The template asks for a conventional commit title, and the title meets that. The description does not fully justify the scope exception, since the linked issue #26083 is not yet triaged, but it does explain the exception and references a maintainer-facing rationale.

Full details: Linked Issues check

Explanation

The changes meet the coding requirements for [#26083, #29899]. ProjectionStore records held_work_ended_at when awaited work stops holding completion. Settlement candidates carry the latest timestamp. ThreadSettlementService applies the hold to incremental, full, and merge sweeps. It waits for a later user message or run request and protects queued report runs during the existing grace period. Migration 80 adds and registers holds_completion and held_work_ended_at. Tests cover the projection transition, lagging-consumer sweeps, queued report runs, and grace expiry.

Full details: Out of Scope Changes check

Explanation

The changes stay within [#26083, #29899]. The projection updates, settlement logic, migration 80, migration registration, migration expectations, and tests implement or verify the requested settlement hold. No unrelated client, contract, setting, manual-settle, or archive behavior changes are shown.

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 918d555 and a132b7e.

Files selected for processing: 1
  • apps/server/src/orchestration-v2/ProjectionStore.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

The hold was recorded in memory from the domain event stream, but full
and merge sweeps read the projection, which can already show the empty
roster while the event consumer lags. The provider thread projection now
records when its roster stops holding work, in the same transaction as
the roster update, and every sweep reads that time from the settlement
candidates. The in-memory tracking is gone.
@dominic-r

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

… is queued

A report requested behind another run stays queued until that run ends, so its request time alone does not mean the report has started. A full or merge sweep in that window could still settle the thread.
@dominic-r

Copy link
Copy Markdown
Contributor Author

@macroscope-app please re-run approvability and correctness on 2c5988e.

@macroscopeapp

macroscopeapp Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Sorry, I'm unable to act on this request because you do not have permissions within this repository.

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor
Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@maria-rcks

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

merged current main without rebasing in 918d555d6b. preserved main's migration 79 and renumbered the held-work migration to 80, updating its registration and migration expectations. no settlement behavior changed during takeover.

verification on blacksmith: vp run check:changed refs/remotes/github/main passed format, lint, knip, graphql, typechecks, and 107 tests across six files. the focused settlement/projection/migration run also passed all 45 tests. two independent reviews approved this head. the three earlier bot findings are already addressed and resolved.

live provider/client reproduction remains unverified; validation used the existing server regression tests. ci is running on the pushed head.

@juliusmarminge juliusmarminge added bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed labels Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Fix PR opened by an issue lead; enters the PR loop pr:gate-failed A land gate failed size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

3 participants