[consensus] slim block codec - #27702
Open
arun-koshy wants to merge 2 commits into
Open
Conversation
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 14, 2026 05:09 — with
GitHub Actions
Active
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 14, 2026 20:30
3ca7b12 to
5473703
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 14, 2026 20:30 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 14, 2026 22:29
5473703 to
0c70f31
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 14, 2026 22:29 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 14, 2026 23:36
0c70f31 to
05e1d6d
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 14, 2026 23:36 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 17, 2026 22:47
05e1d6d to
27179e8
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 17, 2026 22:47 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 18, 2026 05:18
27179e8 to
8dd5e02
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 05:18 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 18, 2026 05:22
8dd5e02 to
849083a
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 05:23 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 18, 2026 05:43
849083a to
96d48a3
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 05:43 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 18, 2026 06:33
96d48a3 to
776d212
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 06:33 — with
GitHub Actions
Active
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 18, 2026 06:36
776d212 to
490b089
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 06:36 — with
GitHub Actions
Active
arun-koshy
marked this pull request as ready for review
August 18, 2026 06:45
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 18, 2026 06:45 — with
GitHub Actions
Active
This was referenced Aug 18, 2026
akichidis
reviewed
Aug 19, 2026
| .map_err(|_| DecodeError::Malformed("claimed_block_digest must be 32 bytes".into()))?; | ||
| let claimed_digest = BlockDigest(claimed_digest); | ||
|
|
||
| let skeleton: Block = bcs::from_bytes(&minimal.block_sans_ancestors) |
Contributor
There was a problem hiding this comment.
why not just block instead of skeleton?
Contributor
Author
There was a problem hiding this comment.
Renamed, though to stripped_block rather than plain block: the rebuild phase has the finished block in scope too, and the point of this local is that it is not a complete block yet (empty ancestors, unverified).
arun-koshy
force-pushed
the
ak/minimal-block-wire-envelope
branch
from
August 20, 2026 02:18
bd00516 to
76b6fef
Compare
Encodes a block's ancestor digests out of the subscription payload and rebuilds them at the receiver. Emission is gated by consensus_slim_block_propagation, which is off on every chain, so nothing changes until it is enabled. A block's ancestors are ~100 references of 32 bytes each, and a receiver at the frontier already holds those blocks. The slim form carries ancestor authors in proposal order plus an override list for the ones the receiver cannot infer -- a non-default round, a slot it cannot resolve uniquely, or an ancestor below the GC horizon. Everything else is looked up locally. Reconstruction is self-validating. The rebuilt bytes must hash to the digest the author claimed before the block goes anywhere, and the signature is checked afterwards on the normal path, so a wrong or hostile digest costs a failed rebuild rather than a bad block. Rebuilding must reproduce the author's exact serialization, since that is what the signature covers. The encoding is produced once at proposal time, where every ancestor is already accepted, so the subscription fan-out shares one encoding rather than each subscriber paying for its own. It rides the Slim arm of the block envelope from the wire change below this one; catch-up replay and the liveness re-broadcast stay on the full form. Decoding is three phases so the state lock covers only the one that needs it. parse_slim validates structure before any lock is taken, so a malformed or misattributed block costs no acquisition. resolve_ancestors runs under the read guard. rebuild_slim reserializes and hashes after it is dropped, keeping a full pass over the block off the lock that accept and commit contend for. Resolution asks DagState only what it needs. Both callers distinguish absent from unique from equivocating, never the full candidate list, so digest_at_slot stops at the second block and reads the digest off the BlockRef key, cloning nothing. A receiver that cannot rebuild a slim block drops it rather than passing it on to be rejected as a peer fault, since the usual cause is an ancestor this node has not accepted yet. Failures are counted by peer and reason in slim_block_decode_failures, and logged by kind: a malformed payload is the peer's fault and stays at info, an unresolvable ancestor is local and drops to debug. The retry counter and backoff still reset, because the stream delivered. An ancestor that does not resolve to exactly one digest falls back to fetching the full block. Choosing between candidates would mean rebuilding and hashing once per combination, and an equivocating peer picks how many candidates there are; the fallback costs one full block, which is what propagation cost before this existed. Two things this deliberately does not do, both required before the flag can be enabled: no emission gate for subscribers trailing the omission horizon (setting the flag in ConsensusProtocolConfig::for_testing makes the authority_node committee tests hang, which is why it stays false there), and no recovery for a block whose ancestors have not arrived yet -- it is dropped and refetched by ordinary block sync.
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 20, 2026 04:09
490b089 to
25bf122
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 20, 2026 04:09 — with
GitHub Actions
Active
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 20, 2026 21:29 — with
GitHub Actions
Active
Naming, per review: the wire payload field is `block` rather than `block_sans_ancestors` (the type already says it carries a block without ancestors), the parsed local is `stripped_block` rather than `skeleton`, `default_round` is `parent_round`, the override local is `ancestor_override` rather than `o`, `rebuild_slim` is `rebuild_from_slim`, and the digest length comes from DIGEST_LENGTH rather than a literal 32. The omission rules move out of `serialize_slim` into `digest_override`, which answers one question -- whether the receiver can resolve this slot itself -- with an early return per rule instead of stacked negations. Each rule now sits next to the reason it exists. Encoding splits along the same seam. `ancestor_overrides` is the only part that reads local state, so `encode` scopes its DagState guard to that call and clones and serializes the block after dropping it. Holding the lock across a full pass over the payload put block size on the critical path of every accept and commit; decoding was already split this way. `ParsedEnvelope` now holds what the later phases actually read rather than the whole decoded message. The encoded payload was retained alongside the decoded block that superseded it, and the claimed digest alongside the copy already extracted from it -- the largest field and a duplicate, kept alive for the whole decode because the struct stored the message wholesale. `ExtendedBlock`'s doc no longer claims the extended information is not forwarded: the slim encoding is the wire payload. It explains instead why this type carries both the block and the encoding while the serialized type carries one form -- the per-subscriber gate chooses from it, and a subscriber served the full form still needs the block.
arun-koshy
force-pushed
the
ak/minimal-block-codec
branch
from
August 20, 2026 21:53
547b2dc to
064530a
Compare
arun-koshy
deployed
to
sui-typescript-aws-kms-test-env
August 20, 2026 21:54 — with
GitHub Actions
Active
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Strips a block's ancestor digests out of the subscription payload and rebuilds them at the receiver. Emission is gated by
consensus_slim_block_propagation, off on every chain, so nothing changes until it is enabled.A block's ancestors are 32 bytes each, and a receiver at the frontier already holds those blocks. The slim form carries ancestor authors in proposal order plus an override for the ones the receiver cannot infer: a non-parent round, a slot it cannot resolve uniquely, or an ancestor below the GC horizon. Everything else is looked up locally.
Reconstruction is self-validating. The rebuilt bytes must hash to the digest the author claimed before the block goes anywhere, and the signature is checked afterwards on the normal path — so a wrong or hostile digest costs a fetch, never a bad block. The rebuild must reproduce the author's exact serialization, since that is what the signature covers.
The encoding is produced once at proposal time, where every ancestor is already accepted, so the whole subscription fan-out shares it. Catch-up replay and the liveness re-broadcast stay on the full form.
Both directions scope their
DagStateguard to the part that needs it: only ancestor resolution reads local state, so cloning, serializing, and hashing the block happen outside the lock that accept and commit contend for.A receiver that cannot rebuild a block drops it rather than passing it on to be rejected as a peer fault, since the usual cause is an ancestor it has not accepted yet. Failures are counted by peer and reason, and logged by kind: a malformed payload is the peer's fault and stays at info, an unresolvable ancestor is local and drops to debug.
Not in this PR, both required before the flag can be enabled
ConsensusProtocolConfig::for_testinghangs theauthority_nodecommittee tests.Stacked on #27695 — review that first; this targets its branch.
Test plan
Fourteen tests on the codec and resolver. The load-bearing ones: byte-identical round trip across all block versions; ancestors carried in proposal order rather than authority order (a decoder that sorted would pass every other test); the GC horizon boundary, where an ancestor sitting exactly on it keeps its digest; a unique-but-wrong local block caught as a digest mismatch; and twelve malformed inputs rejected without panicking.
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.