Conversation
… behind A rollback reverts the database and nothing else. Every cached stored value closes over the last value written through it, so the kernel went on reading the abandoned crank's terminated vats and GC actions, and the next `set` wrote them back. `maybeFreeKrefs` lives only in RAM, so a promise the rollback deleted stayed a collection candidate and the next crank's `collectGarbage` died reading it. Savepoints now carry a snapshot of the candidate set, restored rather than cleared: candidates added while no crank was open are still owed a collection and must survive an unrelated crank's rollback. Closes #1071 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…failed rollback Review follow-up. `clear()` left every cache pointing at rows it had deleted, so the next dequeue killed the run loop — the same defect one function over, now that there is something to call. A failed rollback discards the whole transaction, so RAM goes back to the outermost savepoint rather than the named one. The candidate set is restored first, being the one step of the revert that cannot fail, and `CACHED_VALUES` is keyed by the context's own cached fields so a value added to one and not the other does not compile. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 15, 2026
sirtimid
added this pull request to stack #1108
September 15, 2026 22:38
…caches' into sirtimid/remote-gc-delivery-refused # Conflicts: # packages/ocap-kernel/CHANGELOG.md
The kernel releases its own side of a GC action before telling the endpoint. For a remote that refuses the delivery — or that is simply not connected yet — committing that left the peer holding references this kernel had let go, and its next message naming one of them mints a second kref for the same object, reconciled only by an incarnation change. The crank now aborts, which restores the action and the c-list entries together, and that remote's GC actions are held back until the run loop next has nothing else to do. Holding them back is what makes aborting affordable: GC actions are selected ahead of every other kind of work, so retrying one every crank would starve the kernel to serve one unreachable peer. Waking from an empty queue is both the bound on the wait and the moment when retrying costs nobody anything. A vat that refuses a GC delivery is unchanged: it is broken, and the crank that aborts terminates it. Closes #1064 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up, and the reason for the merge below this commit. Three reviewers found the same thing independently: the abort restores the database row but `gcActions` is a cached stored value, so without the crank's cache revert the action was spent from RAM and the next write erased it from disk too. Aborting was inert. A real-store test now runs the whole sequence and fails without the revert. Also from the reviews: a vat that refuses a GC delivery rethrows before the message about keeping a remote's action is logged; the remote's reachability is read from the same `#lookupEndpoint` as everything else, rather than a second lookup that swallowed every error; and the hold-back test does a crank of real work first, so clearing after every crank no longer passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sirtimid
force-pushed
the
sirtimid/remote-gc-delivery-refused
branch
from
September 16, 2026 16:42
073e07c to
9a57e85
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1098. Its own diff is
git diff sirtimid/endpoint-lookup-and-skip...sirtimid/remote-gc-delivery-refused.The kernel releases its own side of a GC action — clears the reachable flag, or
deletes the c-list entry — before telling the endpoint, because telling an
endpoint to let go is also the kernel letting go. For a vat that refuses, the
crank aborts and the vat is terminated: a local vat that cannot take a GC
delivery is broken. For a remote there is no such answer, and #1022 chose to log
and commit the release anyway, on the grounds that retrying starves the kernel.
The cost of that, found in review and filed as #1064: the peer still holds the
eref. Its next message naming it goes through
exportFromEndpointand mints asecond kref for the same object, and nothing reconciles that until an
incarnation change — for a
retireImports, not even then, sinceforgetEndpointImportskeeps only entries whose direction isexport. #1017describes the frame-flipping half of the same picture.
This is option 1 of the three in the design note: abort and keep the action,
and take remote GC actions off the front of the queue. The crank aborts, which
restores the action and the c-list entries together, and that remote's actions
are held back from selection until the run loop next has nothing else to do.
Holding them back is what makes aborting affordable:
processGCActionSetisconsulted before every other source of work, so re-selecting a failing action
every crank would starve the kernel to serve one unreachable peer. Waking from
an empty queue is both the bound on the wait and the moment when retrying costs
nobody anything.
The same treatment covers a remote the kernel has no handle for at all, which is
every remote until the embedder calls
initRemoteComms. #1098, below this one,deliberately left that case loud rather than dropping what it carried; this is
where it gets somewhere to wait instead.
A note on the merge in this branch
sirtimid/crank-rollback-reverts-caches(#1087) is merged in, because abortingonly keeps the action if
rollbackCrankreverts the cachedgcActionsset aswell as the database row. Without it the abort restores the row, the live kernel
goes on reading the spent set, and the next write erases the row too — the
feature is inert. Three reviewers found this independently and the real-store
test below fails without it. The merge disappears from this diff once #1087
lands on
main.Changes
processGCActionSettakes anisHeldBackpredicate. A held-back endpoint'sactions are not examined at all, which is what keeps them: an action merely
examined is deleted from the durable set whether or not it is selected.
KernelQueue.holdBackRemoteGCand#remotesHeldBack, cleared when the runloop wakes from an empty queue.
KernelRouter.#deliverGCActionaborts and holds the remote back, for a remotethat refuses the delivery and for one it has no handle for. A vat is
unchanged.
@metamask/ocap-kernelchangelog entry under Fixed.Testing
KernelRouter.remote-gc.test.tsis the end-to-end one, against a real nodejsSQLite
:memory:store and the real run loop: a remote refuses aretireImports, and afterwards both the action and the c-list entry are stillthere. Removing the crank's cache revert fails it with
expected [] to strictly equal [ 'r1 retireImport ko1' ], which is what themerge above exists for.
KernelRouter.test.tscovers the three action types against an unreachableremote —
{ abort: true }, nothing released, the remote held back — the refuseddelivery, and a vat that refuses still throwing.
garbage-collection.test.tscovers the predicate: the other endpoints' work is given instead, the held-back
endpoint's actions stay in the durable set, and it gets its turn once nothing
holds it back.
KernelQueue.test.tspins the bound, with a crank of ordinarywork before the queue empties so that clearing after every crank — which is a
livelock — no longer passes.
Six mutations checked in all. Full
@metamask/ocap-kernelsuite green locally;eslint,
constraintsandchangelog:validateclean.Not addressed here, and worth its own change: a held-back remote is not visible
in
KernelStatus, and nothing lifts the hold when the remote reconnects — onlyan idle run loop does. A remote that never comes back is retried whenever the
kernel is idle, forever, with a warning each time.
Closes #1064
🤖 Generated with Claude Code
Note
High Risk
Changes distributed GC delivery, crank abort/rollback, and in-memory store cache consistency—core run-loop paths where mistakes can desynchronize peers or kill the kernel.
Overview
Fixes remote garbage collection so the kernel no longer commits its side of a release when the peer was never told, which could leave the remote holding stale erefs and mint duplicate krefs on the next message.
When a remote has no handle yet or refuses a GC delivery,
KernelRouternow returns{ abort: true }(via#keepForLater) instead of dropping the action or committing after a failed send. The crank rollback restores the spent GC action and c-list state together. Local vats that refuse still throw and terminate as before.To keep retries from starving the run loop (GC actions are picked before all other work),
processGCActionSetaccepts anisHeldBackpredicate,KernelQueuetracks#remotesHeldBack, and held remotes are skipped until the loop parks on an empty queue and clears the set.The same branch includes crank/store fixes needed for abort to work:
rollbackCrankreverts in-memory caches (refreshCachedValues) and restoresmaybeFreeKrefsfrom savepoint snapshots;clear/resetcalldiscardCachedStateso stale run-queue heads cannot kill the next crank.Reviewed by Cursor Bugbot for commit 9a57e85. Bugbot is set up for automated code reviews on this repo. Configure here.