Skip to content

feat(ocap-kernel): keep a GC action a remote could not be told about - #1099

Open
sirtimid wants to merge 7 commits into
sirtimid/endpoint-lookup-and-skipfrom
sirtimid/remote-gc-delivery-refused
Open

sirtimid wants to merge 7 commits into
sirtimid/endpoint-lookup-and-skipfrom
sirtimid/remote-gc-delivery-refused

Conversation

@sirtimid

@sirtimid sirtimid commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1098. Its own diff is
git diff sirtimid/endpoint-lookup-and-skip...sirtimid/remote-gc-delivery-refused.

The kernel releases its own side of a GC action — clears the reachable flag, or
deletes the c-list entry — before telling the endpoint, because telling an
endpoint to let go is also the kernel letting go. For a vat that refuses, the
crank aborts and the vat is terminated: a local vat that cannot take a GC
delivery is broken. For a remote there is no such answer, and #1022 chose to log
and commit the release anyway, on the grounds that retrying starves the kernel.

The cost of that, found in review and filed as #1064: the peer still holds the
eref. Its next message naming it goes through exportFromEndpoint and mints a
second kref for the same object, and nothing reconciles that until an
incarnation change — for a retireImports, not even then, since
forgetEndpointImports keeps only entries whose direction is export. #1017
describes the frame-flipping half of the same picture.

This is option 1 of the three in the design note: abort and keep the action,
and take remote GC actions off the front of the queue.
The crank aborts, which
restores the action and the c-list entries together, and that remote's actions
are held back from selection until the run loop next has nothing else to do.
Holding them back is what makes aborting affordable: processGCActionSet is
consulted before every other source of work, so re-selecting a failing action
every crank would starve the kernel to serve one unreachable peer. Waking from
an empty queue is both the bound on the wait and the moment when retrying costs
nobody anything.

The same treatment covers a remote the kernel has no handle for at all, which is
every remote until the embedder calls initRemoteComms. #1098, below this one,
deliberately left that case loud rather than dropping what it carried; this is
where it gets somewhere to wait instead.

A note on the merge in this branch

sirtimid/crank-rollback-reverts-caches (#1087) is merged in, because aborting
only keeps the action if rollbackCrank reverts the cached gcActions set as
well as the database row. Without it the abort restores the row, the live kernel
goes on reading the spent set, and the next write erases the row too — the
feature is inert. Three reviewers found this independently and the real-store
test below fails without it. The merge disappears from this diff once #1087
lands on main.

Changes

  • processGCActionSet takes an isHeldBack predicate. A held-back endpoint's
    actions are not examined at all, which is what keeps them: an action merely
    examined is deleted from the durable set whether or not it is selected.
  • KernelQueue.holdBackRemoteGC and #remotesHeldBack, cleared when the run
    loop wakes from an empty queue.
  • KernelRouter.#deliverGCAction aborts and holds the remote back, for a remote
    that refuses the delivery and for one it has no handle for. A vat is
    unchanged.
  • @metamask/ocap-kernel changelog entry under Fixed.

Testing

KernelRouter.remote-gc.test.ts is the end-to-end one, against a real nodejs
SQLite :memory: store and the real run loop: a remote refuses a
retireImports, and afterwards both the action and the c-list entry are still
there. Removing the crank's cache revert fails it with
expected [] to strictly equal [ 'r1 retireImport ko1' ], which is what the
merge above exists for.

KernelRouter.test.ts covers the three action types against an unreachable
remote — { abort: true }, nothing released, the remote held back — the refused
delivery, and a vat that refuses still throwing. garbage-collection.test.ts
covers the predicate: the other endpoints' work is given instead, the held-back
endpoint's actions stay in the durable set, and it gets its turn once nothing
holds it back. KernelQueue.test.ts pins the bound, with a crank of ordinary
work before the queue empties so that clearing after every crank — which is a
livelock — no longer passes.

Six mutations checked in all. Full @metamask/ocap-kernel suite green locally;
eslint, constraints and changelog:validate clean.

Not addressed here, and worth its own change: a held-back remote is not visible
in KernelStatus, and nothing lifts the hold when the remote reconnects — only
an idle run loop does. A remote that never comes back is retried whenever the
kernel is idle, forever, with a warning each time.

Closes #1064

🤖 Generated with Claude Code


Note

High Risk
Changes distributed GC delivery, crank abort/rollback, and in-memory store cache consistency—core run-loop paths where mistakes can desynchronize peers or kill the kernel.

Overview
Fixes remote garbage collection so the kernel no longer commits its side of a release when the peer was never told, which could leave the remote holding stale erefs and mint duplicate krefs on the next message.

When a remote has no handle yet or refuses a GC delivery, KernelRouter now returns { abort: true } (via #keepForLater) instead of dropping the action or committing after a failed send. The crank rollback restores the spent GC action and c-list state together. Local vats that refuse still throw and terminate as before.

To keep retries from starving the run loop (GC actions are picked before all other work), processGCActionSet accepts an isHeldBack predicate, KernelQueue tracks #remotesHeldBack, and held remotes are skipped until the loop parks on an empty queue and clears the set.

The same branch includes crank/store fixes needed for abort to work: rollbackCrank reverts in-memory caches (refreshCachedValues) and restores maybeFreeKrefs from savepoint snapshots; clear/reset call discardCachedState so stale run-queue heads cannot kill the next crank.

Reviewed by Cursor Bugbot for commit 9a57e85. Bugbot is set up for automated code reviews on this repo. Configure here.

sirtimid and others added 4 commits September 15, 2026 18:56
… behind

A rollback reverts the database and nothing else. Every cached stored
value closes over the last value written through it, so the kernel went
on reading the abandoned crank's terminated vats and GC actions, and the
next `set` wrote them back. `maybeFreeKrefs` lives only in RAM, so a
promise the rollback deleted stayed a collection candidate and the next
crank's `collectGarbage` died reading it.

Savepoints now carry a snapshot of the candidate set, restored rather
than cleared: candidates added while no crank was open are still owed a
collection and must survive an unrelated crank's rollback.

Closes #1071

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…failed rollback

Review follow-up. `clear()` left every cache pointing at rows it had
deleted, so the next dequeue killed the run loop — the same defect one
function over, now that there is something to call. A failed rollback
discards the whole transaction, so RAM goes back to the outermost
savepoint rather than the named one. The candidate set is restored
first, being the one step of the revert that cannot fail, and
`CACHED_VALUES` is keyed by the context's own cached fields so a value
added to one and not the other does not compile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sirtimid and others added 3 commits September 16, 2026 18:31
…caches' into sirtimid/remote-gc-delivery-refused

# Conflicts:
#	packages/ocap-kernel/CHANGELOG.md
The kernel releases its own side of a GC action before telling the
endpoint. For a remote that refuses the delivery — or that is simply not
connected yet — committing that left the peer holding references this
kernel had let go, and its next message naming one of them mints a
second kref for the same object, reconciled only by an incarnation
change.

The crank now aborts, which restores the action and the c-list entries
together, and that remote's GC actions are held back until the run loop
next has nothing else to do. Holding them back is what makes aborting
affordable: GC actions are selected ahead of every other kind of work,
so retrying one every crank would starve the kernel to serve one
unreachable peer. Waking from an empty queue is both the bound on the
wait and the moment when retrying costs nobody anything.

A vat that refuses a GC delivery is unchanged: it is broken, and the
crank that aborts terminates it.

Closes #1064

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-up, and the reason for the merge below this commit. Three
reviewers found the same thing independently: the abort restores the
database row but `gcActions` is a cached stored value, so without the
crank's cache revert the action was spent from RAM and the next write
erased it from disk too. Aborting was inert. A real-store test now runs
the whole sequence and fails without the revert.

Also from the reviews: a vat that refuses a GC delivery rethrows before
the message about keeping a remote's action is logged; the remote's
reachability is read from the same `#lookupEndpoint` as everything else,
rather than a second lookup that swallowed every error; and the
hold-back test does a crank of real work first, so clearing after every
crank no longer passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A remote's retireImports that fails is never reconciled

1 participant