Why this project exists, what it is trying to be, and the engineering laws it holds to. Read this when
a change feels like it is fighting the grain. docs/ARCH.md is the companion HOW (module map, seams);
this doc is the WHY and the principles everything else cites as "Law N".
An opinionated block-building simulation substrate for Commit-Boost — the layer that stands up a real Ethereum devnet with a relay + builder + commit-boost sidecar, exercises a specific PBS/ePBS feature end to end, and returns a trustworthy verdict you can gate a release on. Nothing here is a mock; the point is not "the devnet booted," it is a verdict.
Mainline ethpandaops/ethereum-package treats out-of-protocol block building as a bespoke, hard-coded
convenience; it has no incentive to keep commit-boost first-class, and ePBS will churn exactly this
surface (relay / builder / sidecar wiring). So cb-testing maintains it, opinionated about
commit-boost, and aims to be the integration layer the release flow never had.
The release loop for a sidecar like this is otherwise: unit tests → ship → wait for testnet feedback →
release. Weak coverage, slow. The integration layer that should sit in the middle is this repo, but the
earlier version was built before test-driven discipline, so it was neither trustworthy nor fast enough
to lean on. As ePBS clients ship, the issue rate rises, and the human-in-the-loop version (paste
terminal output, docker logs a container by hand, isolate the real panic under three masking errors)
does not scale. The win is a loop that can be driven end to end: launch → triage → diagnose → report,
autonomously — legible to a human and to an automated agent from the same output.
The leverage is NOT "rewrite it in Rust." It is three properties, in priority order:
- Preflight against the real images. Validate a rendered config by having the actual relay /
commit-boost image parse it in ~1s, before any 10-minute devnet spend. This is the generalized,
productized form of the manual
docker run --rm <helix:main>loop that first found real schema drift. It is the single biggest legibility win and it retires the whole class of "schema drift discovered 5 minutes into a launch as a masked runtime panic." - Observable by default (not agent-only tooling). Legibility is a system property: structured
tracingevents on everything, a durable verdict report, and AUTOMATIC root-cause capture on any failure, emitted as the normal output of a normal run. A human reads a pretty rendering of that stream; an automated consumer reads the JSON; ONE source of truth, not a separate agent surface. So root-cause capture is a PROPERTY OF THE RUN (when a service dies, the harness attaches its container's root panic to the event stream automatically) — thetriageverb is only the after-the-fact entry point into that same data, not the mechanism. This is strictly better for a human too (root panic inline vsdocker logsby hand). - Feature-asserting scenarios. Every scenario positively asserts its feature's codepath FIRED (skip-sigverify counter > 0, timing-game poll count, extra-validation RPC hit, cross-relay best-bid comparison), not merely "the pipeline didn't crash." A scenario that passes while the feature silently no-oped is a non-test.
Type-reuse from the real config crates is a supporting move (see Law 1), valuable but scoped: it makes commit-boost config drift a compile error; it does NOT solve helix drift (Law 1 caveat). Preflight is what covers both halves.
Non-negotiable; each prevents a named smell. Numbered and quotable — other docs and code cite them as "Law N."
A string-template generator that reverse-engineers a service's serde layout from binary panics is the
root smell and does not survive. Commit-boost config is built from cb_common config structs and
serialized, so a renamed field is a compile error / deny_unknown_fields deserialize error.
Caveat: helix types are NOT reusably importable (divergent branch, different org, a teloxide type
graph), and helix CoresConfig is the field that drifts most. So helix gets an owned, thin
HelixRelayConfig mirror pinned in lockstep with the HELIX_RELAY_IMAGE tag, guarded by the Preflight
law. The few kurtosis-runtime template holes ({{ .Timestamp/.Port/.Network/.Relays }}) are the one
place string-patching survives; isolate them. (Note: because a hand-written mirror is still
hand-mirrored, preflight — not the mirror — is the real guard for helix; see Law 1's interaction with
the config-generation history in docs/ARCH.md §4.)
One image map (no four-way commit-boost/pbs vs commit-boost/commit-boost drift). No checked-in
stale example config that hand-copies generator output; generate on demand instead.
No green from mere block delivery. A pass-gate must key on the signal that proves the feature ran
(e.g. pubkeys_verified, not raw total_events); a best-bid check must compare bid values across
relays, not union-by-slot (one delivering relay must not pass identically to genuine two-relay
aggregation). A harness that lies green is worse than an ugly one.
The checks are pure functions over already-fetched data; inject fixture beacon/relay responses and test the math. Every judgement splits into a pure classifier (data in, verdict out, unit-tested on both sides of every boundary) and a thin I/O shell holding zero verdict logic. CI runs the generator and schema-validates every output.
Everything emits structured tracing events + a durable verdict report; failures auto-attach their
root cause to that stream. Humans and automated consumers tap the SAME surface (pretty rendering vs
JSON) — never a separate agent-only tool category; build good logging on everything and let both
consume it. The VerificationReport is the model to extend, not replace.
ethereum-package is upstream ethpandaops plus a short list of named commits (branch
cb-on-upstream, listed with the reason for each in docs/fork-delta.md). Upstream's own
mev_type: commit-boost is the base; a patch adds only what a scenario needs and upstream cannot
express, or fixes what stock upstream cannot launch.
- (a) Rebase, don't layer. Moving to a new upstream is a rebase of the patch branch plus a full
sweep-gate. A patch upstream has made redundant is dropped, not kept. - (b) A patch earns its place with a failing run and states why upstream is not enough. When the reason stops being true, the patch goes.
- (c) Keep each patch PR-shaped. The bug fixes among them are upstream candidates.
This replaces an earlier plan to carry a (relay, sidecar, builder) abstraction (mev_type: custom)
in a long-lived fork. Measured against current upstream, that abstraction was no longer needed:
upstream's commit-boost type plus a relay-list patch expressed every scenario, and the whole gate
passes on the patch branch. The fork's config shape survives only as the --target fork opt-in.
Scenarios parametrize over EL/CL client pairs. A regression specific to nethermind+prysm is invisible if everything hardcodes geth+lighthouse.
docs/ARCH.md— how the pieces fit (module map, the config↔package seam, the verdict model).docs/CHECKS.md— the per-check catalog (tiers, thresholds, feature-assertion status).docs/DEVELOPING.md— the dev loop + how to add a check or a scenario.docs/fork-delta.md— the patchesethereum-packagecarries on top of upstream, and why each exists.docs/local-kurtosis-e2e.md— the operational runbook.