feat: featureExport / featureConsensusEntropy#693
Draft
sublimator wants to merge 323 commits into
Draft
Conversation
Add missing ttEXPORT/ttCONSENSUS_ENTROPY pseudo transaction fields required by runtime logic and ensure corresponding ledger entries carry threading/sequence fields. Handle ttEXPORT and ttCONSENSUS_ENTROPY in hook stakeholder routing to avoid Unknown transaction type assertion during ledger close.
tequdev
reviewed
Feb 25, 2026
| Env env{ | ||
| *this, | ||
| envconfig(), | ||
| supported_amendments() | featureExportRNG, |
Member
There was a problem hiding this comment.
you can just use supported_amendments()
Suggested change
| supported_amendments() | featureExportRNG, | |
| supported_amendments(), |
| Env env{ | ||
| *this, | ||
| envconfig(), | ||
| supported_amendments() | featureExportRNG, |
Member
There was a problem hiding this comment.
Suggested change
| supported_amendments() | featureExportRNG, | |
| supported_amendments(), |
Collaborator
Author
There was a problem hiding this comment.
I'll need to check this, but note that featureExportRNG adds an extra txn to each ledger for the entropy, so a lot of tests that had exacting transaction related assertions fail with the feature enabled. There was lots of failures, so at the time I moved on and removed it from the default list.
We need a better answer for this, but I wasn't planning to update all the tests, and definitely not just yet.
The merge with origin/dev accidentally stripped all CLOG diagnostic statements from the consensus code path. This restores the clog parameter to internal Consensus.h functions (checkLedger, phaseOpen, closeLedger, updateOurPositions, handleWrongLedger, leaveConsensus, createDisputes) and re-adds all 46 CLOG statements that provide per-round diagnostic detail for phase transitions, convergence progress, dispute tracking, and pause decisions. Also restores the origin/dev structure of Consensus.cpp by removing the anonymous-namespace wrapper and forwarding overloads that were merge artifacts.
The merge with origin/dev accidentally reverted 19 XRPL_ASSERT() calls back to plain assert() and 1 UNREACHABLE() back to assert(0). These macros provide descriptive diagnostic messages on failure and are the project convention since the rippled 2.4.0 migration. Files fixed: - Consensus.h: 9 XRPL_ASSERT reversions - RCLConsensus.cpp: 5 XRPL_ASSERT reversions - BuildLedger.cpp: 3 XRPL_ASSERT reversions - Change.cpp: 1 UNREACHABLE + 1 XRPL_ASSERT reversion
sfHookExportCount was at field code 23, colliding with the mainline rippled UINT16 range. Move to 98 in the Xahau-reserved range. Also reorder sfExportedTxn (90) before sfAmountEntry (91) for consistency.
* Add AMM bid/create/deposit/swap/withdraw/vote invariants:
- Deposit, Withdrawal invariants: `sqrt(asset1Balance * asset2Balance) >= LPTokens`.
- Bid: `sqrt(asset1Balance * asset2Balance) > LPTokens` and the pool balances don't change.
- Create: `sqrt(asset1Balance * assetBalance2) == LPTokens`.
- Swap: `asset1BalanceAfter * asset2BalanceAfter >= asset1BalanceBefore * asset2BalanceBefore`
and `LPTokens` don't change.
- Vote: `LPTokens` and pool balances don't change.
- All AMM and swap transactions: amounts and tokens are greater than zero, except on withdrawal if all tokens
are withdrawn.
* Add AMM deposit and withdraw rounding to ensure AMM invariant:
- On deposit, tokens out are rounded downward and deposit amount is rounded upward.
- On withdrawal, tokens in are rounded upward and withdrawal amount is rounded downward.
* Add Order Book Offer invariant to verify consumed amounts. Consumed amounts are less than the offer.
* Fix Bid validation. `AuthAccount` can't have duplicate accounts or the submitter account.
…ers) - Remove unnecessary cbak() stubs from ConsensusEntropy test hooks and recompile WASM (cbak is optional per Guard.h validator) - Restore RCLCxPeerPos::render() lost during merge (delegates to ConsensusProposal::render()) - Fix Change.cpp applyAmendment() fixInnerObjTemplate2 reversion: use STObject::makeInnerObject() and bracket assignment (fbcff93) - Restore txq-export-quorum-check documentation marker in TxQ.cpp
…change Extract duplicated (n * 80 + 99) / 100 ceiling quorum formula into shared calculateQuorumThreshold() in ConsensusParms.h, matching the standard ValidatorList::calculateQuorum(). Used by ExportSignatureCollector, Change.cpp, and RCLConsensus.cpp. Revert Import.cpp quorum from ceiling back to original truncating formula (totalValidatorCount * 0.8) since Import handles XPOP imports, not the new Export feature. Added TODO for future upgrade.
Add single-sign rejection check in Change::applyExport() matching rippled's multi-sign validation: SigningPubKey must be present but empty, TxnSignature must not be present. Fix Export_test.cpp hook to encode an empty VL blob for SigningPubKey instead of 33 zero bytes (AI slop from export-uvtxn branch).
Due to rounding, the LPTokenBalance of the last LP might not match the LP's trustline balance. This was fixed for `AMMWithdraw` in `fixAMMv1_1` by adjusting the LPTokenBalance to be the same as the trustline balance. Since `AMMClawback` is also performing a withdrawal, we need to adjust LPTokenBalance as well in `AMMClawback.` This change includes: 1. Refactored `verifyAndAdjustLPTokenBalance` function in `AMMUtils`, which both`AMMWithdraw` and `AMMClawback` call to adjust LPTokenBalance. 2. Added the unit test `testLastHolderLPTokenBalance` to test the scenario. 3. Modify the existing unit tests for `fixAMMClawbackRounding`.
Add a generic RuntimeConfig service for runtime-configurable parameters, initially supporting artificial send delays and packet drops for testing consensus behavior on local testnets. - RuntimeConfig class with atomic fast-path gate (zero cost when inactive) - Per-peer targeting via "*" (global) and "ip:port" keys with inheritance - Pre-merged caching at write time for single-lookup read path - Admin RPC handler `runtime_config` (set/clear/clear_all/get) - Env var support: XAHAU_RUNTIME_CONFIG (JSON) or XAHAU_SEND_* vars - PeerImp::send() integration with delay timer and probabilistic drops - RPC handler test covering all operations and merge behavior
- Fix convergence regression caused by 2.4.0 merge: replace stringIsUint256Sized(currenttxhash) with size() < uint256::size() to accept extended proposals (>32 bytes) containing RNG fields - Add message_types filter to RuntimeConfig for targeting specific protocol message categories (proposal, validation, transaction, etc.) - Add appliesTo() method and messageCategories set to ConfigVals - Add category name mapping helpers in RPC handler - Add 2 test cases for message type filtering (8 total)
- Error on unknown message_types instead of silently widening scope - Make messageCategories optional so per-peer can override global filter to "all categories" (nullopt=inherit, empty set=explicitly all) - Clamp send_drop_pct to 0-100% range - Add STARTDIAG: logging for consensus startup diagnostics - Add 3 test cases (11 total, 58 assertions)
Move core xport_reserve and xport implementations from applyHook.cpp DEFINE_HOOK_FUNCTION wrappers into the decoupled HookAPI class, following the same pattern used for etxn_reserve and emit.
Add RNG regression tests for non-UNL data, reveal-without-commit, invalid reveal, and commitment-change stale-reveal handling in CSF consensus tests.
Address review findings on the tier 3 commits: - onPreBuild dedup was type-based (skip if any ttCONSENSUS_ENTROPY present), which silently trusts a pre-present pseudo-tx. Injection is deterministic, so every honest node derives the identical pseudo-tx (identical txID) for the same agreed inputs. Switch to value-based dedup: skip only when the present pseudo-tx EQUALS the one we would produce; a present-but-different entropy pseudo-tx is a determinism violation (version skew / divergent peer) and is now logged at error rather than accepted blindly. (The earlier 'cannot reconstruct the txID locally' justification was wrong — determinism guarantees it can.) - fairRng: read sfEntropyTier defensively (missing => 0 => fail closed). The field is soeREQUIRED so any entry this code wrote carries it; this only guards a pre-tier-3 persisted entry on a long-lived testnet. - quorum_degradation_smoke: assert EntropyTier==consensus_fallback(1) and EntropyCount==0 and non-zero digest explicitly, not just is_fallback.
Follow-up to d6481a3, addressing review of that diff: - The mismatch-log branch in onPreBuild read present pseudo-tx fields (sfDigest/sfEntropyTier/sfEntropyCount) unconditionally — the same throw hazard fairRng was hardened against, here inside onPreBuild during build. Read them defensively (isFieldPresent ? value : '<missing>'). (Note: STTx deserialization enforces all soeREQUIRED fields, so a tier-less ttCONSENSUS_ENTROPY cannot actually reach the set — this is belt-and-suspenders, not a reachable bug.) - Clarify the comment + add action=keep-agreed-and-flag: this is detect-and-log, NOT rejection. The present pseudo-tx is KEPT and still applied at BuildLedger; a hard-fail policy on mismatch is a deliberate future decision (determinism-violation vs halt-risk-under-skew). - Add testOnPreBuildEntropyMismatchKeepsAgreed: a present-but-different entropy pseudo-tx is kept (not replaced), set stays at one entry. - Fix stale 'type-based dedup' comment in the standalone test.
The 'Tier N' preference-order shorthand collides with EntropyTier's strength-ordered values (consensus_fallback=1). Use the enum names in the degradation-smoke comments/descr to remove the ambiguity.
Foundation for Tier 2 (participant_aligned) sub-quorum entropy. No behavior change — nothing consumes these yet; the tier ladder, gates, and selector arrive in later commits. - ActiveValidatorView::originalViewSize: master-key count BEFORE the nUNL subtraction. size() stays the effective (post-nUNL) count used by the 80% validator-quorum gate; originalViewSize is the original-UNL denominator that the 60% Tier 2 floor anchors to, since nUNL can shrink the effective view while leaving faulty nodes in it. - calculateParticipantThreshold(): ceil(0.6 * count), the quorum-intersection floor (two such cohorts always share an honest validator under the ~20% Byzantine bound). - ConsensusExtensions::tier2Threshold(): ceil(0.6 * originalView), anchored to originalViewSize. Tests: originalViewSize asserted on the UNLReport, fallback, and real-ledger nUNL paths; new arithmetic testcase locks ceil(0.6) and the sizing-note boundaries (5->3 banded/unsafe, 6->4 smallest-safe, 8->5 one-nUNL-off).
Collapse the duplicated tier-selection logic in onPreBuild and
buildExplicitFinalProposalTxSet into a single selectEntropy() over the AGREED
entropySetMap_. No behavior change on the production (implicit) path — the
onPreBuild fallback/entropy-set/standalone/mismatch tests pass byte-identically.
Fixes the pre-existing divergence flagged in review: buildExplicitFinalProposalTxSet
derived entropy from local pendingReveals_ while onPreBuild used the agreed
entropySetMap_, so the two could mint different digests for the same round.
Both now share the selector, so the implicit and (experimental, default-off)
explicit-final paths — and any two nodes — derive identical entropy from
identical agreed inputs.
selectEntropy() returns {digest, tier, count} and is a pure function of agreed
round state, so it is directly unit-testable. Injection becomes unconditional:
the selector always yields a fallback digest when there is no validator entropy,
so every RNG-enabled ledger still carries exactly one ConsensusEntropy tx.
Sets up the tier-2 (participant_aligned) ladder, which lands next.
A 60-79% aligned cohort now mints entropy labelled participant_aligned (tier 2) instead of falling back. Healthy >=80% rounds are unchanged (validator_quorum), and a true minority (<60%) still falls back. Mechanics: - entropyGateThreshold() = min(quorumThreshold(), tier2Threshold()): the bar at which the pipeline engages and the entropy conflict gate resolves. In the normal band this is the 60% floor (of the ORIGINAL, pre-nUNL view); under heavy nUNL the band collapses to the 80% quorum and tier 2 vanishes. - selectEntropy() labels the AGREED entropySetMap_ by participant count: >= quorum -> validator_quorum, >= tier2 -> participant_aligned, else fallback. - Tick.h gates (bootstrap-skip, impossible-quorum, commit-timeout, entropy conflict gate) key off entropyGateThreshold() so sub-quorum rounds reach injection instead of short-circuiting. Determinism: the tier LABEL is a function of the agreed set's leaf count (identical on every node holding that hash); the local entropyGateThreshold alignment only decides proceed-vs-fall-back, so divergent local views fall back rather than fork. The 60% floor is over the ORIGINAL view -- the quorum-intersection bound that stops a single equivocator minting two distinct aligned digests under the ~20% Byzantine bound. hasQuorumOfCommits() is deliberately LEFT at 80%: healthy networks keep their exact fast-path and only step down to tier 2 via the commit-timeout path, so this adds zero behavior change above quorum (the degraded band pays one pipeline timeout; a fast-path is a separable follow-up). Folds into featureConsensusEntropy (not yet active), so no new amendment. Tests: new onPreBuild tier-2 case (5-validator view; 4/3/2 revealers -> validator_quorum / participant_aligned / fallback) plus threshold assertions. CSF Peer and the tick test stub mirror entropyGateThreshold() to quorumThreshold() for now -- the end-to-end tier-2 sims (which lower it) land next. Extracted makeUNLReportLedger / harvestCommitReveal test helpers, now shared across the view and harvest tests.
…==0)
calculateParticipantThreshold returned ceil(0.6*n). At every n divisible by 5
(n=5,10,15,20,...) that leaves two aligned cohorts overlapping in exactly
floor(0.2*n) = f validators -- NOT strictly greater than f. So up to f Byzantine
nodes can occupy the entire overlap, leaving no honest validator shared between
the two cohorts, and a single equivocator backed by f-1 colluders can split the
round into two distinct aligned digests -> fork. The spec caught n=5 ("never 5")
but the same failure recurs at every multiple of 5.
Derive the floor from the safety invariant instead: the smallest t with
2t - n > floor(n/5), i.e. floor((n + floor(n/5)) / 2) + 1. This equals
ceil(0.6*n) everywhere except multiples of 5, where it is one higher
(n=10 -> 7, not 6). n=5 now collapses the band (tier2 == quorum), so the
"never 5" operational caveat is enforced by the math rather than a footnote.
Tests: the arithmetic test now asserts the defining invariant
2t - n > floor(n/5) AND t <= quorum for every n in 1..256 (this fails at n=10
under the old formula), plus the bumped boundaries. The onPreBuild tier-2 test
moves off n=5 (no band) onto n=6 (tier2=4, quorum=5; 5/4/3 revealers ->
validator_quorum / participant_aligned / fallback).
Found by Codex adversarial review of the tier-2 implementation.
…l dedup) Two small follow-ups from the Codex review of the tier-2 implementation, both on the consensus-extension internals: - Rename shouldZeroEntropy() -> belowValidatorQuorum(). The selector replaced its callers, leaving it production-dead and MISNAMED: post-tier-2, "below the 80% validator quorum" is no longer "zero entropy" (a participant_aligned set is sub-quorum but non-zero). The new name + a doc comment make it a tier-3 eligibility predicate only and warn against gating injection on it (selectEntropy() owns the tiering). Behavior unchanged. - buildExplicitFinalProposalTxSet now dedups the entropy pseudo-tx by VALUE, not type. Its comment claimed it "mirrors onPreBuild", but onPreBuild went value-based: it verifies a present pseudo-tx is the EXACT txID it would have produced and logs a determinism-violation on mismatch. Explicit-final now does the same (skip-duplicate-verified on match, error log on mismatch), returning the base unchanged either way. Default-off experimental path, so low blast radius, but the two paths now agree.
The CSF peer now mirrors production's tier-2 behavior so the RNG simulations exercise participant_aligned, not just validator_quorum/fallback: - entropyGateThreshold() = min(quorumThreshold(), tier2Threshold()), plus a tier2Threshold() helper (calculateParticipantThreshold over the sim's UNL). - finalizeRoundEntropy() labels by aligned count via the 3-tier ladder (>= quorum -> validator_quorum, >= tier2 -> participant_aligned, else fallback) instead of hardcoding tier 3. The n%5==0 band-collapse keeps the bulk of the suite unaffected: at n=5 tier2 == quorum == 4, so the gate is unchanged for every n=5 and n<=2 network. Only the two n=3 sub-quorum sims cross a live band (n=3: tier2=2 < quorum=3) and now mint tier 2 instead of falling back -- the feature working, with each test's intent preserved: - "impossible quorum fallback" -> "quorum-impossible cohort falls to participant_aligned": 2 of 3 is below 80% but at the tier-2 floor, so it makes progress as tier 2 (no hang, no fork) rather than falling all the way back. - "persistent loss does not shrink quorum": the 2 survivors now mint the labeled-weaker tier 2, NOT tier 3 -- the tier-3 quorum still did not shrink (a min_tier=3 hook rejects it). Intent preserved, outcome relabeled. Other CSF suites (Consensus, ByzantineFailureSim) are unaffected (they do not enable RNG). Next: dedicated tier-2 mint + conflict sims at n=6 (smallest size with a non-degenerate band, f=1).
6 validators is the smallest non-degenerate tier-2 size (f=1, one-wide band {4}: tier2=4, quorum=5). Isolate 2 so the surviving 4-cohort is below the 80% quorum but at the tier-2 floor; it mints participant_aligned entropy (count 4), and all four agree on the same non-zero digest with branches==1 — no hang, no fork. Distributed confirmation of the selector ladder the unit tests already cover.
Codex review caught a CSF/production fidelity gap: production injects entropy from the AGREED entropySetMap_ (selectEntropy), which is frozen at advertise time — late-fetched or conflicting reveals merge into pendingReveals_ but are NOT injected unless a rebuild republishes the hash. The CSF peer fetched into pendingReveals_ AND finalized from pendingReveals_, so a conflict/fetch sim could count reveals production would never inject from. Track the last advertised entropy-set hash (buildEntropySet) and finalize from the sidecar-store snapshot under that hash — the analog of the frozen entropySetMap_. Clean/no-conflict sims are unchanged (the snapshot equals pendingReveals_ there); the model is now faithful for conflict/fetch cases too. Production unaffected (test-harness only). Addresses finding 1 of codex-tier2-final-review.
Codex final-review minor (optional hardening): finalizeRoundEntropy now requires the fetched sidecar entry to be reveal-type before counting it as entropy. lastEntropySetHash_ only ever names a reveal set (hashRngSet's per-type salt rules out a cross-type hash collision), so this is purely defensive — behavior unchanged, all CSF sims green.
Codex final-review minors, both comment-only (no behavior change): EntropyTier participant_aligned no longer says "reserved for a future" tier (it is implemented in this stack); and the commit-timeout path comment said "fixed UNL quorum" but the code now uses entropyGateThreshold() (min(quorum, tier2)).
A 6-node smoke (the smallest NON-degenerate tier-2 size: tier2 floor 4, quorum 5) driving the 4/6 band. n=5 has no band (tier2 == quorum), which is why the existing degradation smoke only ever sees tier 3 / fallback. Tier 2 is below the 80% validation quorum, so the 4/6 cohort's ledgers are provisional: the scenario confirms tier-2 injection from the cohort's logs during the window, then verifies the on-ledger EntropyTier=2 count=4 POST-RECOVERY once those ledgers become canonical and validate (the mechanism the degradation smoke also relies on). Adds the assert_participant_aligned helper and the node_count:6 suite entry. Verified on a live testnet: PASS — 12 tier-2 injections in the 4/6 window, seq 7 & 8 validated as participant_aligned (count 4).
…er check
Codex review minors: assert_validator_quorum at the 5/6 boundary (EntropyTier=3 AND count >= quorum AND non-zero digest, not just tier==3 — catches a bad tier/count pairing); added explicit assert_validator_quorum / assert_consensus_fallback helpers and an entropy_fields() warning that its is_fallback (tier != 3) lumps participant_aligned in with fallback (safe only where no tier-2 band exists).
Robustness: tier 2 is below the validation quorum, so the validated tip stalls and which provisional ledgers it later reaches is timing-dependent (the post-recovery inspection was fragile). Replaced it with direct inspection of the surviving cohort's CLOSED ledger via ledger('closed') DURING the window; recovery is now a pure liveness check.
Verified on a live testnet: PASS — EntropyTier=2 count=4 confirmed on 4 distinct provisional ledgers (seq 7-10).
The 5/6 phase asserted validator_quorum on whichever single ledger the validated tip happened to sit on. But the ledger right at the node-5 drop can be a transient consensus_fallback (EntropyTier=1, count=0) — deterministic and by design, while the commit/reveal pipeline re-primes — so when the tip landed there the assert failed (tier=1, observed ~1 run in 4). Same class of bug as the old post-recovery flake: depending on exactly where the tip lands. Now settle 4 ledgers past the drop, then scan the post-drop validated ledgers for a clean validator_quorum (tier 3, count >= quorum). The window is entirely 5-node cohorts, so a tier-3 there has count == 5 (faithful to '5/6 still tier 3'); the transition fallback is tolerated, not asserted on. Verified 3/3: each run found tier=3 count=5. Also harden _closed_entropy() to raise on != 1 ConsensusEntropy pseudo-tx (mirroring get_entropy_tx) instead of silently skipping — a duplicate/missing injection now fails with a clear error rather than resurfacing as a generic 'no tier-2 ledger'.
tier2Threshold() anchors to originalViewSize (pre-nUNL), not the effective post-nUNL size(): nUNL shrinks the effective view while leaving faulty nodes in it, so a sub-quorum fraction of the effective view can exceed the Byzantine bound. A one-line regression to size() would relabel a forkable cohort as participant_aligned -- and no existing test caught it (the only nUNL test never calls tier2Threshold()/selectEntropy, and the only tier-2 selection test has no nUNL, so original == effective there). Add testTier2ThresholdAnchorsToOriginalView: 8 active validators, 2 disabled via NegativeUNL (original 8, effective 6), asserting tier2Threshold()==5 (from the original 8) not 4 (from the effective 6), with quorum/gate cross-checks. Mutation-verified: flipping the production read to size() turns exactly this test red (tier2Threshold + entropyGateThreshold) while the existing tier-2 selection test stays green. Also: pointer comment at tier2Threshold() linking the invariant to its guard test, and a design-doc fix -- the worked example used n=5, where the band is empty (quorum == participant_aligned == 4) so it illustrated an unreachable tier; now n=6.
The tier-2 anchor pin (20d52d8) used an 8/6 (original/effective) nUNL config whose rationale was wrong: it computed cohort overlap against the original view (2*4-8=0), but aligned cohorts form in the EFFECTIVE view, so two 4-of-6 cohorts overlap by 2*4-6=2 > floor(8/5)=1 -- 8/6 is NOT actually forkable, the original-view anchor is merely conservative there. The test still caught the originalViewSize->size() regression, but the stated reason was misleading (review catch). Switch to 10 active / 2 disabled (original 10, effective 8), a genuinely forkable case: a 5-of-8 cohort (what the effective-size threshold would admit) overlaps only 2*5-8=2, which does NOT exceed the f=floor(10/5)=2 faulty the original UNL still tolerates -> an equivocator could mint two distinct tier-2 digests. The correct original anchor requires 7 (overlap 6 > 2) and keeps the band closed (tier2==quorum==7); a regression to size() drops the floor to 5 and re-opens the forkable [5,7) band. Assertions and rationale updated to match; suite green (897, 0 failures).
# Conflicts: # src/xrpld/app/hook/applyHook.h
…(F1) The entropy/export bless-vs-fallback gate counted alignment over ALL tx-converged trusted proposers (currPeerPositions_), while the thresholds and the entropy leaf set are computed over the active validator view. A node that locally trusts proposers outside the on-ledger UNLReport active set could pad alignedParticipants(), inflating the counting universe N above originalViewSize and eroding the Tier-2 intersection margin (2t - N) below the Byzantine floor f -- so two equivocation cohorts padded by non-active aligners could each clear the gate (review finding F1). Backstopped by the 80% validation quorum, but the proof's universe and the code's universe must match. inspectTxConvergedSidecarPeers now takes an active-view membership predicate and counts only member peers toward alignment; the local +1 is gated on localIsActiveValidator(). Both the RNG entropy gate and the export-sig gate pass ext.isUNLReportMember / ext.localIsActiveValidator, mirroring buildEntropySet / hasQuorumOfCommits' containsNode filter. New method on ConsensusExtensions + CSF Peer + the FakeExtensions stub. Regression test (Sidecar peer alignment helper): a trusted-but-non-active aligned proposer pushes unfiltered aligned to 2 but filtered aligned stays 1, and a non-active local node's +1 is suppressed -- padding cannot satisfy the gate. ConsensusExtensions 900, ConsensusRng 386, Consensus 1399, all green. Note: the explicit-final proposal path (ConsensusExtensionsTick.h ~:934) counts over prevProposers and is NOT yet filtered (separate, experimental path).
testInvalidEntropyRequirements rejects four invalid dice/random requirements then returned accept(0,0,0) on success. But a valid dice(6,..) returns 0..5, so a regression that let the min_tier=0 requirement through (returning a value that happened to be 0) would still pass ~1/6 of the time -- the weakest spot is exactly the min_tier lower bound it most needs to prove (review finding). Return sentinel 42 (distinct from any dice/random result and from INVALID_ARGUMENT) and assert ==42, so any leaked requirement returns its own non-42 code and fails. WASM block recompiled; ConsensusEntropy 138 tests, 0 failures. Follow-up to F1 (fb9e271).
The explicit-final proposal path counts participants/alignment over the unfiltered trusted-proposer set (ctx.peerPositions) plus an unconditional local +1, not the active validator view -- the same gap the F1 fix closed for the main entropy gate (inspectTxConvergedSidecarPeers). It is default-off/experimental with no robust timing model found (per the existing TBD, may never ship); if it is ever enabled it must apply the same active-view membership filter. Comment-only. Follow-up to F1 (fb9e271).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Last updated: 2026-06-17 | branch: feature-export-rng | commit: c8fad50 (Merge origin/dev into feature-export-rng)
featureExport + featureConsensusEntropy: Cross-Chain Exports + Decentralized Secure Randomness
This PR introduces two features under separate amendment flags:
featureExport— Cross-chain transaction export: hooks or users create transactions signed by the network's validators for use on external chainsfeatureConsensusEntropy— Consensus-derived randomness: deterministic, manipulation-resistant entropy available to hooks viadice()andrandom()Both share validator-set infrastructure and quorum calculation, but are independently amendment-gated.
Amendment Relationship
The two amendments are independent but complementary:
Neither blocks the other: CE's RNG sub-states and Export's sig convergence run in parallel during the establish phase. Both check their own gates independently and fall back gracefully if they can't converge.
The quorum rationale behind this matrix is covered once, in Tiered Quorum below.
UNL Source and Fallback Behavior
RNG uses UNLReport ActiveValidators (
UNLReport.sfActiveValidators) as the canonical validator set for non-fallback entropy labels.The validator view has two modes:
validator_quorumandparticipant_alignedRNG labels. Thresholds are ledger-anchored; the tier-2 floor is anchored to the original pre-NegativeUNL report size.consensus_fallbackentropy, even on a healthy local testnet, because local validator configuration is not a cross-node deterministic tier-label authority.Part 1: Cross-Chain Transaction Export
What This Feature Does
This feature enables cross-chain transaction export — allowing a hook or user on Xahau to create a transaction that Xahau's validators collectively sign. The resulting multisigned transaction is a normal, valid transaction on XRPL — no protocol changes or cooperation from XRPL are required.
How it works from XRPL's perspective
An account on XRPL is set up with a SignerList pointing to Xahau validator keys. When Xahau's validators sign an exported transaction, they're effectively acting as members of that SignerList. Once enough signatures are collected, the transaction can be submitted to XRPL as a standard multi-signed transaction. XRPL doesn't know or care that the signatures came from Xahau.
This design requires zero XRPL-side changes, no XLS specification, and no amendment on XRPL.
How it works from Xahau's perspective
A hook calls
xport()or a user submits attEXPORTtransaction (type 91). The transaction enters the open ledger (tesSUCCESSprovisional), and validators attach their multisign signatures to consensus proposals. When enough validators have signed (quorum met), the export succeeds withsfExportResultin the transaction metadata — containing the fully multisigned transaction assfExportedTxn(readable JSON, ready for raw submission to XRPL). If quorum isn't reached beforeLastLedgerSequence, the export expires withtecEXPORT_EXPIRED.In standalone mode (unit tests / dev), Export::doApply signs directly with the node's own validator keys — no consensus proposals needed.
Exported transactions must use
TicketSequence(withSequence=0) because a bounced transaction on the destination chain would jam sequential sequence numbers. ANetworkIDguard rejects exports targeting the local network or from unconfigured nodes.Export Architecture
Exports use a retriable transaction pattern with proposal-based signature collection:
ttEXPORT→ enters the open ledger withtesSUCCESS(provisional, consumes sequence + fee)sfExportedTxn), computes its multisign signature (buildMultiSigningData+sign), and attachestxHash(32) + pubkey(33) + signature(~72)toTMProposeSet.exportSignatures(deduplicated per round viamarkSent())ConsensusExtensions::onTrustedPeerMessage(), sender-bound to the proposal key, and stored inExportSigCollectorsfExportedTxninsidesfExportResultmetadata, creates shadow ticket keyed by the signed tx hash (getHash(HashPrefix::transactionID)— includes Signers)terRETRY_EXPORT(retained for next ledger, no fee/sequence consumed)tecEXPORT_EXPIRED(sequence consumed, clean failure — preventstefMAX_LEDGERsilently dropping the tx on the next ledger)Only the final result touches the ledger. No intermediate signature accumulation. The multisigned blob in metadata is ready for raw submission to XRPL. On a healthy network an export typically completes in the same ledger it was submitted in.
Tiered Quorum: Safety Without Complexity
Export quorum adapts based on whether ConsensusEntropy is also enabled.
Without CE, export signature collection is ephemeral: different validators may see different sig counts at ledger close, and an 80% threshold could cause validators to flip between success and retry across ledger boundaries — acceptable but not ideal. Unanimity avoids that churn entirely, at the cost of any missing validator blocking the export.
With CE, Export piggybacks on CE's
ExtendedPositionserialization, sub-state machine, and SHAMap fetch/merge infrastructure to converge on a shared sig set before closing the ledger. All validators agree on exactly which exports have quorum, enabling the standard 80% threshold.ExportSigCollectorcalculateQuorumThreshold)ExtendedPosition, sub-state machine, and SHAMap fetch/merge for deterministic agreementSHAMap Convergence (with CE)
When CE is enabled, export sigs converge using the same infrastructure as RNG commit/reveal:
exportSigSetHashadded toExtendedPosition(flag 0x10)SHAMapType::SIDECARmapsSTObject(sfGeneric)sidecars tagged withsfSidecarType=sidecarExportSigtnSIDECAR/wireTypeSidecar, are content-addressed withHashPrefix::sidecar, and are keyed bysidecar.getHash(HashPrefix::sidecar)sfSidecarTypeis the payload discriminator for export sigs, RNG commits, and RNG reveals; the hash prefix stays generic to the sidecar leaf class instead of splitting by payload kindSignature Collection
Proposal Attachment
Each validator scans the open ledger for
ttEXPORTtxns, extracts the inner tx (sfExportedTxn), computes its multisign signature viabuildMultiSigningData(), and attachestxHash + pubkey + signatureto the proposal. Gated onfeatureExport, deduplicated per round viamarkSent().Proposal attachment code (ConsensusExtensions.cpp)
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:2336-2450Sig Harvesting
PeerImpverifies and routes trusted proposal messages; export signature parsing and storage happen inConsensusExtensions::onTrustedPeerMessage(). Each wire entry is variable-length:txHash(32) + pubkey(33) + signature(~72). The embedded pubkey must match the proposal sender, and proposal-level trust is checked before any signature is stored.When the matching
ttEXPORTis present in the open ledger, the multisign signature is verified immediately and stored as verified. If relay ordering means the transaction is not available yet, the signature is stored as unverified and can be upgraded later by the transactor path after reconstructing the inner transaction.ExportSigCollector
Thread-safe collector (
ExportSigCollector.h) with stale cleanup (256-ledger timeout). It separates verified signatures from unverified proposal-ingress signatures. Only verified signatures count toward quorum, appear in sidecar SHAMaps, and are assembled into the multisigned transaction. Key methods:addVerifiedSignature(txHash, pubkey, signature)— stores a cryptographically verified multisign signatureaddUnverifiedSignature(txHash, pubkey, signature)— stores a trusted but not-yet-verified proposal signatureupgradeSignature(txHash, pubkey, verifiedBuf)— promotes a matching unverified signature after verificationcheckQuorumAndSnapshot(txHash, threshold)— atomically checks verified quorum and returns the verified bufferssnapshotWithSigs()— returns verified signatures for sidecar-set constructionmarkSent(txHash)— deduplicates per consensus roundcleanupStale(ledgerSeq)— removes entries older than 256 ledgersExport Transactor
The transactor has three outcomes on the closed ledger:
sfExportResultmetadataterRETRY_EXPORT(before LLS) ortecEXPORT_EXPIRED(on the LLS ledger — last chance, sequence consumed cleanly)Key detail: the shadow ticket stores
getHash(HashPrefix::transactionID)of the multisigned blob (which includes the Signers array), not the unsigned inner tx hash. This ensures the hash matches what ends up in the XPOP when the tx executes on XRPL.Export::doApply (full source)
📍
src/xrpld/app/tx/detail/Export.cpp:89-295Shadow Tickets and the Export Round-Trip
Export is a 3-way handshake, not fire-and-forget:
sfExportedTxninsidesfExportResultmetadata — readable JSON, ready for raw submissionltSHADOW_TICKET) is created, keyed by account + ticket sequence, storing the signed tx hash (getHash(HashPrefix::transactionID)— includes all fields including Signers)SignerListpoints to the Xahau validator keys, so XRPL validates it as a standard multisigned transaction. It executes (or bounces), producing an XPOP.ttIMPORT. Import checks the shadow ticket exists, verifies the XPOP inner tx hash matches the shadow ticket's stored hash, consumes the shadow ticket (frees reserve), and fires hooks.Shadow tickets are round-trip completion tokens:
xport_cancel()hook API orsfCancelTicketSequenceonttEXPORTttEXPORTtransactionWhen
ttIMPORTseessfTicketSequenceon the inner transaction, it takes the export callback path: verify shadow ticket hash match, consume the shadow ticket, fire hooks, done. The export callback path skips thesfOperationLimitand signing key match checks (the shadow ticket already proves the relationship). No B2M balance crediting. When there's nosfTicketSequence, it takes the existing Burn-to-Mint path unchanged.Hook Integration
Hooks call
xport()which internally constructs attEXPORTwrapper (withsfEmitDetails) and pushes it onto the emitted txn queue. The wrapper flows through the normal emitted txn path:xport_reserve(N)— reserves N export slots (also reserves emit slots)xport(inner_tx_blob)— validates inner tx → constructsttEXPORTwrapper → emits itttEXPORTenters the open ledger next round → proposal-based sig collection → retriable transactorThe hook receives the inner tx hash (the cross-chain transaction it built), while the
ttEXPORTwrapper handles the Xahau-side lifecycle.Export Protocol Additions
sfExportResult(OBJECT 98)sfExportedTxn+sfLedgerSequence+sfTransactionHash)sfCancelTicketSequence(UINT32 101)sfExportedTxn(OBJECT 90)ltSHADOW_TICKET(0x5374)ttEXPORT(91)terRETRY_EXPORTtecEXPORT_EXPIRED(200)TMProposeSet.exportSignatures(field 13)xport(),xport_reserve(),xport_cancel()Part 2: Decentralized Secure Randomness
Adding randomness to deterministic consensus sounds simple until you try to do it without breaking safety. This part implements Same-Ledger Usable Randomness: finalizing entropy after user intent is locked, but before normal execution in that same ledger.
Review Scope
featureConsensusEntropyisDefaultNo; behavior is inert until enabled by amendment vote.ConsensusEntropy_test(Hookdice()/random(), fallback semantics),ConsensusExtensions_test(the establish-phase tick gates: commit/reveal/entropy states, export sig convergence, sidecar build/fetch/merge, active validator view),ConsensusRng_testandExtendedPosition_test(sub-state transitions, serialization compatibility, malformed wire cases), plus the export and ingress suites (Export_test,ExportSigCollector_test,ExportSignatureHarvester_test,ExportSignatureUpgrader_test,ExportResultBuilder_test,XportWrapperBuilder_test,ProposalPrecheck_test). NotefeatureConsensusEntropyis deliberately excluded from the defaultjtx::Envfeature set (see Test Environment Gating); CE behavior is exercised by its dedicated suites.How It Works
The architecture centers on converging on signed input sets rather than voting on a derived output hash. This ensures that every node can independently verify and reconstruct the final result.
1. Transport: Piggybacked Proposals
The
ConsensusProposalwire format is extended viaExtendedPosition. Most entropy data (commitments and reveals) flows through existing proposal gossip with low incremental payload overhead on the fast path, while consensus latency cost comes from the added sub-state progression/timeouts.ExtendedPosition::operator==only compares thetxSetHash. RNG sub-state differences never stall the core consensus on user transactions.2. Pipelined Sub-states
RNG progression runs inside internal
establishsub-states. These are checkpoints within the existing consensus cadence:ConvergingTx: Normal transaction convergence while harvesting entropy commitments.ConvergingCommit: Locking thecommitSeton the healthy 80% fast path, with bounded timeout paths so degraded rounds do not stall consensus.ConvergingReveal: Targets reveals from 100% of known committers, bounded by timeout/fallback paths (including the 1.5s reveal timeout) to preserve liveness. Final tier selection happens later from the agreed entropy set, not from local pending reveals.3. SHAMap Union Convergence
Harvested commitments and reveals are stored in ephemeral, unbacked SHAMaps.
InboundTransactionspipeline to fetch only the missing leaves from peers.4. Synthetic Injection & Same-Ledger Execution
Once reveals are collected, the final entropy is computed deterministically (
sha512Half(sorted_reveals)).buildLCL(Ledger Construction), the node locally synthesizes attCONSENSUS_ENTROPYpseudo-transaction.Hook API Integration
Provides two new deterministic WebAssembly APIs for Hook developers:
dice(sides, min_tier, min_count): Returns an integer in[0, sides-1].random(write_ptr, write_len, min_tier, min_count): Fills a buffer (up to 512 bytes) with consensus-derived randomness.Entropy is tiered, and the quality requirement is a required argument — there is deliberately no default. Every ledger carries entropy at one of three tiers, recorded on the
ConsensusEntropyledger entry assfEntropyTier+sfEntropyCount:validator_quorum(3)EntropyCount= contributing validatorsparticipant_aligned(2)consensus_fallback(1)H(prefix, parentLedgerHash, baseTxSetHash, seq)); unpredictable in practice but user-influenceable via tx submission — never for value-bearing outcomes.EntropyCount = 0Both APIs gate on the caller's stated requirement: entropy is served iff it is fresh (current or previous ledger) and
tier >= min_tier && count >= min_count; otherwise the call fails closed withTOO_LITTLE_ENTROPY.min_tiermust be one of the stored tiers1..3, andmin_countmust fit the on-ledgerUINT16EntropyCount; invalid requirements returnINVALID_ARGUMENT. A lottery passes(min_tier=3, min_count=N)and refuses degraded/fallback rounds; a lower-value game may pass(2, M)and accept participant-aligned degraded entropy; a cosmetic shuffle may pass(1, 0)and accept any fresh entropy — but that acceptance is explicit and reviewable at the call site. WASM imports have no default parameters, so the alternative to required arguments is a hidden network constant invisible at every call site (the previous design hard-codedcount >= 5).Per-hook output streams are domain-separated: each call mixes the ledger entropy with the ledger seq, txn ID, originating account, hook hash, hook account, chain position, weak/strong execution flag, and a per-execution call counter, so no two hook executions (or successive calls within one) observe the same stream.
One known UX caveat (flagged as a TODO in
fairRng): during open-ledger (speculative) execution, the entropy for the in-flight ledger does not exist yet, so the APIs draw on the previous ledger's entropy — while final execution during ledger close uses the current ledger's freshly injected entropy. A hook'sdice()/random()results can therefore differ between provisional and final application. This is inherent to same-ledger entropy finalization (entropy is locked after user intent, so it cannot be visible to the speculative pass); only the closed-ledger result is canonical, and hook authors should treat provisional outcomes the way they already treat provisionaltesSUCCESS.fairRng (hook-facing entropy derivation)
📍
src/xrpld/app/hook/detail/applyHook.cpp:4049-4120Safety & Liveness
participant_alignedis intentionally below validation-quorum strength and therefore opt-in. Hooks that require fail-closed validator-quorum entropy keep passingmin_tier=3; hooks that can tolerate provisional/degraded entropy must opt intomin_tier=2.Adversarial Considerations (Entropy Quality)
The safety claim above is about ledger agreement. Entropy quality under adversarial behavior is a separate axis, and the commit/reveal design bounds — but does not eliminate — validator influence:
reveal-without-commitment).sha512Half(reveal, pubKey, seq) == commitment, verified at harvest and again at merge.sfBlob); fetched reveal leaves are bound to a trusted validator identity (NodeID ↔ trusted key) and verified against the on-record commitment. Entries from non-UNL senders are rejected at every boundary.participant_alignedorconsensus_fallback. These transitions are labeled and bounded; neither lets an attacker choose an output value at the validator tier. The fallback tier itself is user-influenceable (a quiet-ledger submitter can grind the tx set), which is why it is a distinct labeled tier that hooks must opt into.This makes the entropy suitable for applications where outcomes must be unpredictable and manipulation must be bounded and observable. Applications needing stronger guarantees against validator collusion (e.g., very high-value lotteries) should layer additional mechanisms on top.
Infrastructure & Support Logic
Several non-obvious plumbing changes were required to make the RNG pipeline robust and testable:
1. Fast Polling during RNG Transitions
To reduce the latency impact of the extra sub-states, the heartbeat timer accelerates to 250ms (tunable via
XAHAU_RNG_POLL_MS) while in the RNG pipeline.📍
src/xrpld/app/misc/NetworkOPs.cpp:1032-10502. Local Testnet Resource Charging
Connections from
127.0.0.1normally share a single IP resource bucket. This change preserves the port for loopback addresses so that local multi-node testnets don't hit peer resource limits due to the increased RNG set traffic.📍
include/xrpl/resource/detail/Logic.h:113-1173. Test Environment Gating
featureConsensusEntropyis excluded from defaultjtx::Envtests to prevent its automatic pseudo-tx injection from breaking existing test suites that rely on specific transaction counts.📍
src/test/jtx/Env.h:86-894. Sidecar Set Sync Filtering
Internal RNG and Export sidecar data is stored as
STObject(sfGeneric)leaves in ephemeralSHAMapType::SIDECARmaps.InboundTransactionsstill provides the fetch machinery, but sidecar acquisition usesSidecarSetSFinstead ofConsensusTransSetSF, so fetched leaves are cached as SHAMap nodes and are never parsed or submitted as transactions.📍
src/xrpld/app/ledger/detail/TransactionAcquire.cpp:47-56📍
src/xrpld/app/ledger/SidecarSetSF.cpp:28-40📍
src/xrpld/app/ledger/SidecarSetSF.cpp:42-50Guided Code Review (Projected Source)
This section follows runtime order so the code reads as a story, not a file dump.
1) Proposal payload:
ExtendedPositioncarries RNG + Export sidecar fieldsExtendedPositionadds commit/reveal set identities, export sig set hash, and per-validator leaves while keeping tx-set identity explicit.Non-obvious:
operator==compares onlytxSetHashon purpose. That decouples core tx-set convergence from RNG/Export sub-state drift.operator==(equality firewall):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:114-149add()(signed serialization of all sidecar fields):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:163-206fromSerialIter()(legacy + extended wire decode):📍
src/xrpld/app/consensus/RCLCxPeerPos.h:233-2822) Harvest stage: trust boundary + reveal verification
Incoming RNG data is rejected for senders outside the active validator view, and reveals are accepted only if they match prior commitments. Non-fallback entropy labels require that active view to be UNLReport-backed; the trusted-config fallback view can collect data but mints only
consensus_fallback.📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1850-19813) Quorum basis: active UNL snapshot; expected proposers are liveness hints
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:207-220📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:271-3264) State-machine checkpoints:
ConvergingTx -> ConvergingCommit -> ConvergingReveal📍
src/xrpld/consensus/ConsensusExtensionsTick.h:94-10665) Export sig convergence gate (parallel with RNG)
📍
src/xrpld/consensus/ConsensusExtensionsTick.h:1070-13156) Sidecar SHAMap construction: commit proofs, deterministic reveal leaves, export sig sidecars
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:652-714📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:722-782📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:786-8437) Injection stage (A): final entropy selection with deterministic fallback
📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1710-17198) Injection stage (B): build and enqueue
ttCONSENSUS_ENTROPY📍
src/xrpld/app/consensus/ConsensusExtensions.cpp:1728-18239) Build stage: entropy pseudo-tx executes before normal transactions
📍
src/xrpld/app/ledger/detail/BuildLedger.cpp:111-14810) Apply stage: write consensus entropy into the singleton ledger object
📍
src/xrpld/app/tx/detail/Change.cpp:248-26511) Wire anchor: proposal message carrying extended payload bytes
📍
include/xrpl/proto/ripple.proto:153-177