Transaction Chaining

[aszepieniec]

Very much work-in-progress. Please see TODO.md for a motivation and progress tracker.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing asz/transaction-chaining (92cb17f) with master (28ff10f)}

[aszepieniec]

Transaction chaining — benchmark results

neptune-consensus/benches/chaintx.rs; cargo bench -p neptune-consensus --bench chaintx.

Measured on my i9-12900K (16 cores / 24 threads, 123 GB RAM) with TVM_LDE_TRACE=no_cache.

Claim 1 — Forge + Fix beats Collect + Raise

One transaction, 2 in / 2 out / 1 announcement, same primitive witness down both routes.

	legacy	chaining
Σ padded height	2 183 168	1 572 864
prover time	858 s	599 s

breakdown

  chained: Forge -> Fix
    Forge[0]  height  1048576     422.1 s
    Fix       height   524288     177.1 s
    TOTAL     height  1572864     599.2 s

  legacy: Collect -> Raise
    RemovalRecordsIntegrity  height    65536      12.1 s
    CollectLockScripts       height     4096       0.5 s
    KernelToOutputs          height     8192       0.8 s
    CollectTypeScripts       height     8192       0.7 s
    Raise                    height  2097152     844.1 s
    TOTAL                    height  2183168     858.2 s

≈30% cheaper.

Claim 2 — chaining N interactions beats N separate transactions

Not like-for-like: the chained fixture's successor spends its predecessors' unconfirmed outputs, which the legacy pipeline cannot express at all. The closest counterpart is N independent transactions raised and merged.

N = 2	legacy	chaining
Σ padded height	5 414 912	3 670 016
prover time	2 119 s	1 361 s

breakdown

  chained: 2 interactions
    Forge[0]  height  1048576     392.5 s
    Forge[1]  height  1048576     391.7 s
    Chain[0]  height  1048576     397.5 s
    Fix       height   524288     178.9 s
    TOTAL     height  3670016    1360.6 s

  legacy: 2 separate transactions
    RemovalRecordsIntegrity[0]  height    65536      11.5 s
    CollectLockScripts[0]       height     4096       0.5 s
    KernelToOutputs[0]          height     8192       0.9 s
    CollectTypeScripts[0]       height     8192       0.8 s
    Raise[0]                    height  2097152     839.9 s
    RemovalRecordsIntegrity[1]  height    65536      11.5 s
    CollectLockScripts[1]       height     4096       0.5 s
    KernelToOutputs[1]          height     8192       0.7 s
    CollectTypeScripts[1]       height     8192       0.8 s
    Raise[1]                    height  2097152     850.4 s
    Merge[0]                    height  1048576     401.7 s
    TOTAL                       height  5414912    2119.2 s

≈36% cheaper. At the margin: one more chained interaction is Forge + Chain ≈790 s, against ≈1265 s for one more legacy transaction.

Why

Not the RRI inlining: proving RRI alone is 12.1 s, 1.4% of the legacy total. It is recursive verification which dominates all of these programs. Forge+Fix nets 3 fewer recursive verifications than Raise.

N = 4 compiles but has not been run. no_cache makes both columns pessimistic in absolute terms; the ratio is unaffected.

[Sword-Smith]

Thanks for the benchmark. Very cool that this is faster than "collect" + "raise"!

As you mention, all that really matter is the number of in-vm verifications you have to do. Since "collect" alone (without the "raise" step) is still much faster than the "chaining" approach, I think that "collect" should stay as the default proving method that downstream wallets use. That being said, that's of course up to each wallet implementation and outside of our control.

[Sword-Smith]

The auditing of the entire witness might be a bit excessive. I wonder what you pay for that in clock cycles. I'm thinking about this TASM:

let audit_preloaded_data =
            library.import(Box::new(VerifyNdSiIntegrity::<ChainWitness>::default()));

Maybe it's OK? I think it's worth investigating what the witness auditing costs. We don't want that to become dominant in the future, for any transactions sizes. Consider using more "WitnessMemory" structures if auditing is too expensive.

[Sword-Smith]

Let me suggest that we don't call the existing transaction pipeline the "legacy" pipeline, since that communicates that it will be retired, which is not the case. Let's instead call it the "single-proof" pipeline.

Just posting a stream of consciousness here. Maybe it'll be useful for others.

Vocabulary

• confirmed input An input to a LinkTx that comes from the mutator set. So a UTXO that was previously mined, and that is being spent in this LinkTx.
• thruputs Inputs to a LinkTx that comes from outputs from another LinkTx.

The reason that "update" on a transaction with no inputs was disallowed, was that the "update" of a "nop" transaction would grant composers a fast way to mine empty blocks. With the current ban on updating a transaction with no inputs, composers wanting to mine an empty block are required to create a "nop" transaction from scratch for each new block proposal, going through "raise" for both their coinbase transaction, and their "nop" transaction. Including a transaction from the mempool in their block proposal requires the composer only to go through one "raise" and an "update", as opposed to two "raise".

Let's ensure that composers keep this incentive, to include mempool transactions in their block proposals.

New Transaction Pipeline

[Sword-Smith]

So we need to ban "update" on LinkTx when the LinkTx has no confirmed inputs and no thruputs.

And we should ban negative fees throughout the LinkTx pipeline. Those are only required in single proof's "merge" operation.

Also: The SingleProof program digest could be printed to std out when it is used to construct the SingleProof claim inside of the LinkTxProof program.

edit:
Alan agrees on:

• ban "update" in chain pipeline when confirmed_inputs.len() + thruputs.len() == 0.
• Ban negative fees throughout the chain pipeline, transitively banned through entrypoints "cast" and "forge"
• Agree on the security-in-depth/redundant check that prints the single proof program digest in all the link proof executions. This changes the claim for the LinkProof program.

[Sword-Smith]

Updated to latest triton-vm and tasm-lib version: v8.0.0.

[Sword-Smith]

Missing internal audits:

LinkProof

• Forge
  • a. coinbase is None
  • b. merge bit is false
  • c. fee is non-negative
  • d. All confirmed inputs live in the AOCL and match the absolute indices in the witness data -- AOCL membership / absolute index set correspondence
  • thruputs ++ confirmed inputs == input UTXO witness data
  • outputs == output UTXO witness data
  • All output elements match UTXOs from witness field outputs
  • NativeCurrency typescript is always required. Also if inputs, thruputs, and outputs are empty.
  • Type script list constructed correctly
  • All type scripts halt
  • Lock script list constructed correctly
  • All lock scripts halt
  • no bfe-wrap when summing number of thruputs and number of (confirmed) inputs.
  • All fields MAST verified before use
    • fee (during c)
    • coinbase (a)
    • merge bit (b)
    • AOCL MMR from witness (intial verification)
• Chain
• Advance
  • a. Previous LinkProof is valid for the relevant link-kernel MH
  • b. no coinbase
  • c. merge-bit not set
  • d. new_timestamp >= old_timestamp
  • e. disallow empty thruput + empty inputs
  • f. Fee unchanged
  • g. announcements unchanged
  • h. outputs unchanged
  • i. mutator set moved forward
  • j. All promoted thruputs are members of the new AOCL
  • k. Promoted thruputs inputs are calculated from the same witness data as the verified AOCL leafs
  • l. All promotions have strictly increasing AOCL leaf indices
  • m. new thruputs are old throughputs minus promotions <=> set(old thruputs) = set(new thruputs) ++ set(promotions)
  • n. new inputs are old inputs plus promotions
  • o. promotion-witness loop iteration only uses bounded jumps.
  • All fields MAST verified before use
    • old mutator set hash / old AOCL <- old lkmh (during i)
    • new mutator set hash new AOCL -< new lkmh (i)
    • old inputs (j/k/l/m)
    • new inputs (j/k/l/m)
    • old thruputs (j/k/l/m)
    • new thruputs (j/k/l/m)
    • old outputs (h)
    • new outputs (h)
    • old announcements (g)
    • new announcements (g)
    • new fee (f)
    • old fee (f)
    • old timestamp (d)
    • new timestamp (d)
    • new coinbase (b)
    • new merge bit (c)
  • Non-MAST authenticated fields don't need to be authenticated
    • old coinbase (Forge and Cast block coinbase from being anything but None)
    • old merge-bit (as above)
• Cast
  • disallow set coinbase field
  • disallow set merge bit
  • force fee to be non-zero
  • claimed tx-kernel MH matches link-kernel MH in claim (with empty thruputs set)
  • SingleProof for tx-kernel is valid

SingleProof

• Fix

Everything else is window dressing.