Skip to content

[consensus] slim block codec - #27702

Open
arun-koshy wants to merge 2 commits into
ak/minimal-block-wire-envelopefrom
ak/minimal-block-codec
Open

[consensus] slim block codec#27702
arun-koshy wants to merge 2 commits into
ak/minimal-block-wire-envelopefrom
ak/minimal-block-codec

Conversation

@arun-koshy

@arun-koshy arun-koshy commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Description

Strips a block's ancestor digests out of the subscription payload and rebuilds them at the receiver. Emission is gated by consensus_slim_block_propagation, off on every chain, so nothing changes until it is enabled.

A block's ancestors are 32 bytes each, and a receiver at the frontier already holds those blocks. The slim form carries ancestor authors in proposal order plus an override for the ones the receiver cannot infer: a non-parent round, a slot it cannot resolve uniquely, or an ancestor below the GC horizon. Everything else is looked up locally.

Reconstruction is self-validating. The rebuilt bytes must hash to the digest the author claimed before the block goes anywhere, and the signature is checked afterwards on the normal path — so a wrong or hostile digest costs a fetch, never a bad block. The rebuild must reproduce the author's exact serialization, since that is what the signature covers.

The encoding is produced once at proposal time, where every ancestor is already accepted, so the whole subscription fan-out shares it. Catch-up replay and the liveness re-broadcast stay on the full form.

Both directions scope their DagState guard to the part that needs it: only ancestor resolution reads local state, so cloning, serializing, and hashing the block happen outside the lock that accept and commit contend for.

A receiver that cannot rebuild a block drops it rather than passing it on to be rejected as a peer fault, since the usual cause is an ancestor it has not accepted yet. Failures are counted by peer and reason, and logged by kind: a malformed payload is the peer's fault and stays at info, an unresolvable ancestor is local and drops to debug.

Not in this PR, both required before the flag can be enabled

  • No per-subscriber gate. The flag gates emission globally, but nothing here decides full-vs-slim per peer: a subscriber trailing the horizon cannot rebuild anything and must be served full blocks. Observable rather than theoretical — setting the flag in ConsensusProtocolConfig::for_testing hangs the authority_node committee tests.
  • No recovery for a block whose ancestors have not arrived yet; it is dropped and refetched by ordinary block sync.

Stacked on #27695 — review that first; this targets its branch.

Test plan

Fourteen tests on the codec and resolver. The load-bearing ones: byte-identical round trip across all block versions; ancestors carried in proposal order rather than authority order (a decoder that sorted would pass every other test); the GC horizon boundary, where an ancestor sitting exactly on it keeps its digest; a unique-but-wrong local block caught as a digest mismatch; and twelve malformed inputs rejected without panicking.


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC:
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 14, 2026 05:09 — with GitHub Actions Active
@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sui-docs Ready Ready Preview Aug 20, 2026 9:57pm
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
multisig-toolkit Ignored Ignored Preview Aug 20, 2026 9:57pm
sui-kiosk Ignored Ignored Preview Aug 20, 2026 9:57pm

Request Review

@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 3ca7b12 to 5473703 Compare August 14, 2026 20:30
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 14, 2026 20:30 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 5473703 to 0c70f31 Compare August 14, 2026 22:29
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 14, 2026 22:29 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 0c70f31 to 05e1d6d Compare August 14, 2026 23:36
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 14, 2026 23:36 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 05e1d6d to 27179e8 Compare August 17, 2026 22:47
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 17, 2026 22:47 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 27179e8 to 8dd5e02 Compare August 18, 2026 05:18
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 18, 2026 05:18 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 8dd5e02 to 849083a Compare August 18, 2026 05:22
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 18, 2026 05:23 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 849083a to 96d48a3 Compare August 18, 2026 05:43
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 18, 2026 05:43 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 96d48a3 to 776d212 Compare August 18, 2026 06:33
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 18, 2026 06:33 — with GitHub Actions Active
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 776d212 to 490b089 Compare August 18, 2026 06:36
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 18, 2026 06:36 — with GitHub Actions Active
Comment thread consensus/core/src/network/tonic_network.rs Outdated
Comment thread consensus/core/src/slim_block.rs Outdated
Comment thread consensus/core/src/slim_block.rs Outdated
Comment thread consensus/core/src/minimal_block.rs Outdated
Comment thread consensus/core/src/minimal_block.rs Outdated
.map_err(|_| DecodeError::Malformed("claimed_block_digest must be 32 bytes".into()))?;
let claimed_digest = BlockDigest(claimed_digest);

let skeleton: Block = bcs::from_bytes(&minimal.block_sans_ancestors)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just block instead of skeleton?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed, though to stripped_block rather than plain block: the rebuild phase has the finished block in scope too, and the point of this local is that it is not a complete block yet (empty ancestors, unverified).

Comment thread consensus/core/src/slim_block.rs Outdated
Comment thread consensus/core/src/slim_block.rs Outdated
Comment thread consensus/core/src/minimal_block.rs Outdated
Comment thread consensus/core/src/minimal_block.rs Outdated
Comment thread consensus/core/src/slim_block.rs Outdated
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-wire-envelope branch from bd00516 to 76b6fef Compare August 20, 2026 02:18
Encodes a block's ancestor digests out of the subscription payload and rebuilds
them at the receiver. Emission is gated by consensus_slim_block_propagation,
which is off on every chain, so nothing changes until it is enabled.

A block's ancestors are ~100 references of 32 bytes each, and a receiver at the
frontier already holds those blocks. The slim form carries ancestor authors in
proposal order plus an override list for the ones the receiver cannot infer -- a
non-default round, a slot it cannot resolve uniquely, or an ancestor below the
GC horizon. Everything else is looked up locally.

Reconstruction is self-validating. The rebuilt bytes must hash to the digest the
author claimed before the block goes anywhere, and the signature is checked
afterwards on the normal path, so a wrong or hostile digest costs a failed
rebuild rather than a bad block. Rebuilding must reproduce the author's exact
serialization, since that is what the signature covers.

The encoding is produced once at proposal time, where every ancestor is already
accepted, so the subscription fan-out shares one encoding rather than each
subscriber paying for its own. It rides the Slim arm of the block envelope from
the wire change below this one; catch-up replay and the liveness re-broadcast
stay on the full form.

Decoding is three phases so the state lock covers only the one that needs it.
parse_slim validates structure before any lock is taken, so a malformed or
misattributed block costs no acquisition. resolve_ancestors runs under the read
guard. rebuild_slim reserializes and hashes after it is dropped, keeping a full
pass over the block off the lock that accept and commit contend for.

Resolution asks DagState only what it needs. Both callers distinguish absent
from unique from equivocating, never the full candidate list, so digest_at_slot
stops at the second block and reads the digest off the BlockRef key, cloning
nothing.

A receiver that cannot rebuild a slim block drops it rather than passing it on
to be rejected as a peer fault, since the usual cause is an ancestor this node
has not accepted yet. Failures are counted by peer and reason in
slim_block_decode_failures, and logged by kind: a malformed payload is the
peer's fault and stays at info, an unresolvable ancestor is local and drops to
debug. The retry counter and backoff still reset, because the stream delivered.

An ancestor that does not resolve to exactly one digest falls back to fetching
the full block. Choosing between candidates would mean rebuilding and hashing
once per combination, and an equivocating peer picks how many candidates there
are; the fallback costs one full block, which is what propagation cost before
this existed.

Two things this deliberately does not do, both required before the flag can be
enabled: no emission gate for subscribers trailing the omission horizon (setting
the flag in ConsensusProtocolConfig::for_testing makes the authority_node
committee tests hang, which is why it stays false there), and no recovery for a
block whose ancestors have not arrived yet -- it is dropped and refetched by
ordinary block sync.
@arun-koshy
arun-koshy force-pushed the ak/minimal-block-codec branch from 490b089 to 25bf122 Compare August 20, 2026 04:09
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 20, 2026 04:09 — with GitHub Actions Active
@arun-koshy arun-koshy changed the title [consensus] minimal block codec [consensus] slim block codec Aug 20, 2026
@arun-koshy
arun-koshy deployed to sui-typescript-aws-kms-test-env August 20, 2026 21:29 — with GitHub Actions Active
Naming, per review: the wire payload field is `block` rather than
`block_sans_ancestors` (the type already says it carries a block without
ancestors), the parsed local is `stripped_block` rather than `skeleton`,
`default_round` is `parent_round`, the override local is `ancestor_override`
rather than `o`, `rebuild_slim` is `rebuild_from_slim`, and the digest length
comes from DIGEST_LENGTH rather than a literal 32.

The omission rules move out of `serialize_slim` into `digest_override`, which
answers one question -- whether the receiver can resolve this slot itself --
with an early return per rule instead of stacked negations. Each rule now sits
next to the reason it exists.

Encoding splits along the same seam. `ancestor_overrides` is the only part that
reads local state, so `encode` scopes its DagState guard to that call and clones
and serializes the block after dropping it. Holding the lock across a full pass
over the payload put block size on the critical path of every accept and commit;
decoding was already split this way.

`ParsedEnvelope` now holds what the later phases actually read rather than the
whole decoded message. The encoded payload was retained alongside the decoded
block that superseded it, and the claimed digest alongside the copy already
extracted from it -- the largest field and a duplicate, kept alive for the whole
decode because the struct stored the message wholesale.

`ExtendedBlock`'s doc no longer claims the extended information is not forwarded:
the slim encoding is the wire payload. It explains instead why this type carries
both the block and the encoding while the serialized type carries one form --
the per-subscriber gate chooses from it, and a subscriber served the full form
still needs the block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants