M5 epic: multi-chain deploy + packaging + docs reconciliation#72
Closed
brunota20 wants to merge 114 commits into
Closed
M5 epic: multi-chain deploy + packaging + docs reconciliation#72brunota20 wants to merge 114 commits into
brunota20 wants to merge 114 commits into
Conversation
Adds the dependencies the 0.2 host backends need: - cowprotocol (1.0.0-alpha) for the cow-api submission path (OrderBookApi, OrderCreation, OrderUid, Chain). - alloy-provider / -rpc-client / -transport-ws / -primitives (1.5) for the chain JSON-RPC dispatch. The reqwest feature on alloy-provider engages connect_http; the pubsub/ws features back eth_subscribe-class methods. - redb (2) for local-store. Same crate cowprotocol's own watch-tower picked, so the dep tree does not bifurcate when both are used in the same workspace. - reqwest (0.12, rustls-tls) — direct, so the import survives any future cowprotocol feature rearrangement. - tracing + tracing-subscriber (env-filter + fmt) — replaces the 0.1 eprintln! debug log so the engine can drop into a structured log pipeline without re-instrumenting every host call. - thiserror (2) — typed error enums in each backend. - tempfile + wiremock as dev-deps for the host backend tests. Adds engine.example.toml documenting the [engine] state_dir + per- chain RPC URLs the chain backend reads at boot; data/ is now ignored so a local run does not leave the redb file in tree.
Replaces the 0.2 Unsupported stubs with working backends. Each
capability lives in its own host submodule so the trait impls in
main.rs stay thin (dispatch + project the backend's typed error
onto HostError).
cow_api::submit_order
- Parses the guest's bytes as JSON cowprotocol::OrderCreation.
- Dispatches via cowprotocol::OrderBookApi::post_order.
- Returns the assigned OrderUid as a 0x-prefixed hex string.
cow_api::request
- REST passthrough. The base URL is whichever URL the pool's
OrderBookApi client carries — so OrderBookApi::new_with_base_url
overrides (staging, wiremock) flow through transparently.
- Method/path validated host-side; orderbook 4xx/5xx bodies are
surfaced verbatim so the guest can decode {errorType,description}.
chain::request
- Raw JSON-RPC dispatch over an alloy DynProvider opened from
engine.toml at boot. WebSocket URLs engage pubsub (eth_subscribe);
HTTP URLs use the HTTP transport. Params are passed as
serde_json::RawValue so alloy does not re-encode.
- request-batch falls back to per-call dispatch (same shape as the
earlier stub but now backed by real RPC).
local_store
- redb file under engine_config.engine.state_dir.
- Single shared table. Per-module namespacing is enforced
host-side via [len:u8][module_name][raw_key] prefix on every
key. list_keys strips the prefix before returning to the guest.
logging
- Routes through tracing::event! tagged with module=<namespace>.
- Engine boot installs an EnvFilter-based subscriber; RUST_LOG
overrides the engine.toml log_level.
identity / remote-store / messaging / http stay at Unsupported per
the 0.2 roadmap (keystore / Swarm / Waku land in 0.3).
Tests (14, all green):
- cow_orderbook: pool default chains, unknown-chain typing, REST
GET passthrough, relative-path resolution, unknown-method
rejection, submit_order round-trip — last three under wiremock
so the full HTTP path is exercised without hitting api.cow.fi.
- provider_pool: empty pool surfaces UnknownChain.
- local_store: roundtrip, namespace isolation, delete, list_keys
prefix-stripping, empty-namespace rejection.
End-to-end against modules/example: example.wasm loads under the
new wiring, logs init + on_event through the tracing pipeline.
… death (BLEU-813-817)
…er-pool, supervisor (BLEU-821)
…interfaces (BLEU-819)
…ed_crate_dependencies, drop redundant map_err)
PR #9 specific: - main: warn + return when block/log streams end (WebSocket dropped) - supervisor: simplify dispatch_block by extracting chain_id before move - supervisor: temp_local_store returns (TempDir, LocalStore) instead of leaking - README: correct engine.toml chain syntax to [chains.<id>] with rpc_url Rebased from PR #8: - local_store_redb: table.range() instead of iter() for O(matching) keys - provider_pool: dedupe method clone on the success path - main: hex_encode writes into the pre-allocated buffer - cow_orderbook: drop blank line nit - manifest: collapse nested if and use ? operator (clippy) - alloy_rpc_client / alloy_transport(_ws) imports as _ to satisfy unused_crate_dependencies.
Move the manifest.rs monolith into a directory module with four focused submodules (types, load, capabilities, error). Includes the Subscription enum and the four PR #9 tests for subscription parsing. Behaviour unchanged - pure code motion.
main.rs went from 739 lines of mixed bootstrap + 8 Host trait impls +
CLI parser + event loop to ~125 lines of pure orchestration. New
layout:
- bindings.rs: wasmtime::component::bindgen!() moved out so other
modules can name the generated types.
- cli.rs: Cli struct + manual parser.
- host/state.rs: HostState + WasiView impl.
- host/error.rs: unimplemented / internal_error / hex_encode helpers.
- host/impls/{chain,cow_api,identity,local_store,remote_store,messaging,
logging,clock,random,http,types}.rs: one Host trait impl per file.
- runtime/limits.rs: DEFAULT_FUEL_PER_EVENT + DEFAULT_MEMORY_LIMIT.
- runtime/event_loop.rs: open_block_streams, open_log_streams, run,
wait_for_shutdown_signal, TaggedBlockStream, TaggedLogStream.
Adding a new capability is now a single new file under host/impls/
rather than a 60-80 line diff in main.rs.
local_store_redb.rs was 89% tests, cow_orderbook.rs was 60%, and supervisor.rs was 32% (205 lines absolute). Promote each to a directory module with the test suite living in a sibling tests.rs so impl-side diffs stop competing with test churn for attention.
Carries PR #8 (host backends) + PR #9 (supervisor) + cowprotocol patch. Open upstream: nullislabs#15.
Open upstream: nullislabs#12. Resolved .gitignore by taking the PR #12 additions (.agents/, .claude/, skills-lock.json) plus PR #15's data/. # Conflicts: # .gitignore
Add modules/twap-monitor/ as a workspace member. Cargo.toml declares [lib] crate-type = ["cdylib"] for WASM Component output, and pulls the deps the TWAP module path needs: cowprotocol (default-features off — only typed primitives and OrderCreation surface needed), alloy-sol-types (event/return decoding lands in BLEU-826/827), and wit-bindgen. src/lib.rs binds against the shepherd:cow/shepherd world (event- module imports + cow-api). generate_all is required because the world include pulls nexum:host/types across packages — without it, wit-bindgen panics on the missing cross-package mapping. init and on_event are stubbed: init logs once; on_event is a no-op until the Event::Log / Event::Block dispatch lands in BLEU-826 / BLEU-827. Verification: cargo build --target wasm32-wasip2 --release -p twap-monitor emits a 65 KB .wasm. Engine load is gated on module.toml (BLEU-834).
…-826)
`on_event(Event::Logs)` decodes each log against
`ComposableCoW.ConditionalOrderCreated` via `alloy_sol_types`,
extracts `(owner, params)`, and writes `watch:{owner}:{params_hash}`
to local-store with the abi-encoded `ConditionalOrderParams` as
the value. BLEU-827 reads this back via `list-keys("watch:")` and
the value is exactly the `(handler, salt, staticInput)` tuple the
poll path passes to `getTradeableOrderWithSignature`.
Idempotency: `local_store::set` overwrites in place, so re-org
replay or overlapping subscription windows produce no observable
side effect.
Resilience: `decode_conditional_order_created` returns `None`
when topic0 does not match the event signature or the payload
fails ABI decoding. Adjacent events on the same subscription
(MerkleRootSet, SwapGuardSet) are silently skipped instead of
short-circuiting the batch. The fn is on plain slices so the
host-free unit tests cover well-formed / wrong-topic / empty-
topics without wit-bindgen scaffolding.
Block, Tick, and Message variants of `Event` are left unhandled
in this PR — `Event::Block` dispatch lands in BLEU-827 (poll
path); the other two are not used by this module.
Adds `alloy-primitives` as a direct dep so the topic/data plumbing
does not rely on alloy types leaking through `cowprotocol`'s
re-exports.
`cargo build --target wasm32-wasip2 --release -p twap-monitor`
emits a 96 KB .wasm (up from the 65 KB skeleton because of the
alloy + cowprotocol composable types now linked in).
`on_event(Event::Block)` walks every persisted watch, skips the
ones gated by a future `next_block:` / `next_epoch:` entry, and
dispatches the ready ones via `chain::request("eth_call",
[{to: COMPOSABLE_COW, data}, "latest"])` to
`ComposableCoW.getTradeableOrderWithSignature(owner, params,
"", [])`.
Returns:
- Successful return data → `<(GPv2OrderData, Bytes)>::abi_decode_params`
→ `PollOutcome::Ready { order, signature }`.
- Revert payload → `decode_revert` matches the four-byte selector
against the five `IConditionalOrder` errors:
OrderNotValid → DontTryAgain
PollNever → DontTryAgain
PollTryNextBlock → TryNextBlock
PollTryAtBlock(n) → TryOnBlock(n)
PollTryAtEpoch(t) → TryAtEpoch(t)
- Anything else falls back to TryNextBlock so a flaky RPC or
unmodelled require-revert is retried instead of dropped.
Decoder ABI: a local `abi::Params` struct mirrors the wire format
of `cowprotocol::ConditionalOrderParams` because sol! cannot cross
crate boundaries; the resulting call selector is byte-equal to the
real contract. The successful return path decodes into the
canonical `cowprotocol::GPv2OrderData` directly, so the 12-field
struct is not duplicated. `Ready` boxes the order to keep
`PollOutcome` cache-friendly (clippy::large_enum_variant).
Storage conventions (shared with BLEU-830, which writes these):
- `next_block:{owner}:{params_hash}` -> u64 LE — block number gate
- `next_epoch:{owner}:{params_hash}` -> u64 LE — Unix-seconds gate
Either / both / neither may be set; the watch polls when both pass.
`block.timestamp` is milliseconds per WIT, so we divide by 1000 to
compare against the `TryAtEpoch` (seconds) convention.
Host follow-up: the chain backend currently swallows alloy's
`RpcError::ErrorResp.data` (it becomes `host-error.message`,
unstructured). `poll_one` is wired to consume structured revert
hex via `host-error.data` once that lands — the `decode_revert_hex`
test locks the path. Until then, every revert defaults to
TryNextBlock, which is the safe choice.
Tests: 14 new (return round-trip, all five revert variants, hex
plumbing, eth_call JSON shape, watch-key round-trip, U256
saturation), keeping the 3 BLEU-826 regressions. `.wasm` grows
from 96 KB to 215 KB (serde_json + IConditionalOrder ABI + the
GPv2OrderData decode path linked in).
Linear: BLEU-827. Ref ADR-0006.
…828)
On `PollOutcome::Ready { order, signature }`, convert the
`GPv2OrderData` to the typed `OrderData` (maps the on-chain
bytes32 markers `kind` / `sellTokenBalance` / `buyTokenBalance`
via cowprotocol's `from_contract_bytes`), wrap the signature as
`Signature::Eip1271` (ComposableCoW returns the orderbook wire
form: raw verifier bytes, the orderbook re-prepends `from`
before settlement), and feed everything through
`OrderCreation::from_signed_order_data`. The body is then
serde-encoded and pushed to `cow_api::submit_order(chain_id,
body)`.
On success, persist `submitted:{uid}` in local-store as an
empty marker — presence of the key is the receipt; BLEU-830
may later attach metadata but the bare flag is enough to
suppress double submits.
Scope notes (deliberately deferred):
- `app_data` is hard-coded to `EMPTY_APP_DATA_JSON`.
Conditional orders that pin a real document on IPFS get
rejected by `from_signed_order_data` (digest mismatch) and
skipped with a Warn log instead of submitting a corrupt body.
Resolving the document is its own concern.
- Submission errors are logged. BLEU-829 wires
`OrderPostError::retry_hint` into this site so the backoff /
drop decision is data-driven.
- `from` is set to the watch owner (the address that emitted
`ConditionalOrderCreated`). The orderbook prepends this to
the EIP-1271 blob during settlement.
Tests: 7 new (gpv2_to_order_data marker mapping incl. zero-
receiver normalisation, unknown kind / balance marker
rejection; build_order_creation happy path with serde round-
trip; rejection of non-empty app_data and `from = ZERO`).
Total 24 host tests. `.wasm` 273 KB (was 215 KB; serde for
OrderCreation, the OrderData/Signature/SigningScheme modules,
and serde_with's runtime ride along).
Linear: BLEU-828. Ref ADR-0006 (modules build orders
themselves).
Captures the 2026-06-18 COW-1064 dry run + live in-flight validation of PR #47 (resolve_app_data fix). ## Acceptance summary 5 of 6 rows green; the only [ ] is `block delta ≥ 1500` (got 415) because the run was intentionally interrupted twice to validate PR #47 against the same data/e2e local-store across pre-PR-47 + PR-47-twap-monitor + PR-47-ethflow-watcher commits. | Row | Result | |---|---| | block delta ≥ 1500 | [ ] (got 415; 3 engine restarts for PR #47 mid-run validation) | | all 5 modules have a terminal marker | [x] | | shepherd_module_errors_total{trap} == 0 | [x] | | no module poisoned | [x] | | 0 ERROR lines from nexum_engine | [x] | | TWAP + EthFlow tx submitted | [x] | ## 4 anomalies filed in Linear, fully documented in §6 - COW-1074 — twap-monitor + ethflow-watcher hardcoded EMPTY_APP_DATA_JSON. **Fixed in-run via PR #47**; live-validated for both modules (§6.5). - COW-1075 — SDK classify_api_error should map `DuplicatedOrder` -> `Drop` (stop-loss retry loop). - COW-1076 — ethflow on-chain `validTo=uint32::MAX` rejected by Sepolia orderbook (`ExcessiveValidTo`; upstream issue). - COW-1077 — scripts/e2e-onchain.sh TWAP `t0=0` produces permanently-finished order (caller-side encoding bug). ## Live PR #47 validation (§6.5 — the key methodology note) Three engine binaries exercised on the same redb local-store: 1. `5bcd47b` (pre-PR-47): surfaces the digest-mismatch client-side skip for both twap-monitor + ethflow-watcher on non-empty appData orders. 2. `acc9654` (PR #47 twap-monitor): existing cow-swap UI TWAP re-polls to Ready -> resolve_app_data resolves the JSON from `/api/v1/app_data/{hash}` -> submit reaches orderbook -> DuplicatedOrder (server-side reject only). Client-side digest check bypassed. 3. `cd68de0` (PR #47 ethflow-watcher): new cow-swap UI EthFlow swap (`0x82da5ced...`) observed -> appData = `0xe46e7d0c...` (NON-empty rich JSON: appCode="CoW Swap", slippageBips=857, smartSlippage=true) -> resolve_app_data calls orderbook -> JSON extracted from `fullAppData` field -> build produces matching-digest body -> submit reaches orderbook -> ExcessiveValidTo (server-side reject only, tracked separately in COW-1076). The PR #47 fix is therefore live-validated end-to-end against the real Sepolia orderbook in **both** affected modules. ## What this report unblocks COW-1031 (7-day soak) is technically unblocked: the engine + 5-module dispatch is proven correct under live conditions; PR #47 closes the only blocking SDK gap for the soak's TWAP + EthFlow coverage. The remaining 3 follow-ups (COW-1075/1076/1077) are quality-of-output rather than correctness regressions and do not block the soak. Operator sign-off pending in §8. Linear: COW-1064 (closes).
…COW-1075)
`OrderBookPool::submit_order_json` returns `CowApiError::Orderbook(cowprotocol::Error::OrderbookApi { status, api })` for any 4xx with a typed `{"errorType": "...", ...}` body (see `cowprotocol::transport::HttpResponse::into_status_error`). The WIT adapter was dropping `api` on the floor (`data: None`), so the guest's `shepherd_sdk::cow::classify_api_error` always saw `None` and fell back to its safe-default `TryNextBlock`. Permanent rejections like `DuplicatedOrder`, `InvalidSignature`, or `ExcessiveValidTo` therefore looped forever, masquerading as transient failures.
Root cause of the stop-loss infinite-retry behaviour observed in the 2026-06-18 COW-1064 dry run (e2e-report-2026-06-18.md §6.3): 76 retries of an already-submitted order in 170 blocks because the host never let the guest see what the orderbook actually said.
Fix is in the WIT adapter (`crates/nexum-engine/src/host/impls/cow_api.rs`), not the SDK classifier. The classifier already handles `Unknown(_)` -> `Drop` correctly via its `Some(_) => Drop` branch; it just needed the envelope to dispatch on. Extracted the projection into a testable `orderbook_to_host_error` helper that:
- serialises `ApiError` into `HostError.data` as JSON when the variant is `OrderbookApi { status, api }` (the only variant carrying a structured payload),
- sets `code` to the HTTP status so guests can disambiguate 4xx vs 5xx,
- leaves `data: None` for other `cowprotocol::Error` variants (transport, serde, unexpected-status) since they have no envelope and `TryNextBlock` is the correct safe default for them.
Tests:
- `orderbook_to_host_error` unit tests cover the envelope-forward, the optional inner `data` round-trip, and the non-envelope `UnexpectedStatus` branch (3 cases).
- New wiremock integration test `submit_order_propagates_orderbook_envelope` confirms a 400 with `errorType: "DuplicatedOrder"` surfaces the `OrderbookApi` variant end-to-end through `OrderBookPool::submit_order_json`.
All 13 cow-api-adjacent tests pass; workspace tests untouched.
…1076)
EthFlow on-chain orders use `validTo = u32::MAX` by design (see `cowprotocol::eth_flow`). The Sepolia orderbook's max-validTo cap rejects this shape with `errorType = "ExcessiveValidTo"`, and after the COW-1075 host fix the strategy already classifies it correctly as Drop. The remaining gap was operator ergonomics: every EthFlow placement on Sepolia produced a Warn-level "ethflow dropped" line, which would dominate a 7-day soak dashboard with non-anomalous traffic.
Change: in `apply_submit_retry`'s Drop arm, peek at the decoded ApiError. If the orderbook's `errorType == "ExcessiveValidTo"`, log at Info instead of Warn. All other Drop reasons (InvalidSignature, WrongOwner, etc.) keep Warn so real anomalies still page the operator. Dispatch (write `dropped:{uid}`, clear stale `backoff:{uid}`) is unchanged.
Why not gate on more (e.g. inspect the order's validTo field): the strategy already filters logs to EthFlow contract addresses; ExcessiveValidTo from the orderbook for an EthFlow placement is unambiguously the documented constraint. Keeping the gate narrow avoids accidentally suppressing other-cause Warns.
Tests (3 new in `modules/ethflow-watcher/src/strategy.rs`):
- `submit_excessive_valid_to_logs_at_info_not_warn`: end-to-end through `on_logs`; confirms exactly one drop line at Info level and zero Warn drops for this case.
- `submit_other_permanent_error_still_logs_at_warn`: regression guard - InvalidSignature stays at Warn.
- `submit_drop_without_envelope_keeps_warn_level`: predicate-level unit test confirming `is_expected_excessive_valid_to` returns false when `HostError.data` is None (e.g. transport failure).
Docs: added "Known upstream constraints on Sepolia" section to `docs/operations/e2e-testnet-runbook.md` documenting this gap, the post-fix operator-visible behaviour, the Prometheus signal (`shepherd_cow_api_submit_total{outcome=\"err\"}` grows by the EthFlow placement count then stops), and a pointer to COW-1076 for the upstream-confirmation status.
Soak impact: the COW-1031 7-day run on Sepolia will now show ExcessiveValidTo drops as Info-level traffic. The soak's "0 unexpected errors" acceptance bar is preserved because Warn-level drops only fire on real anomalies.
All 17 ethflow-watcher tests pass (+3 new); workspace tests untouched. clippy + fmt clean. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
The previous `e2e-onchain.sh` pinned a 516-byte hex blob with `t0 = 0` in the ComposableCoW.create() static-input tuple. TWAP handler's `validateData` does NOT reject `t0 = 0` (it only checks `t0 >= type(uint32).max`), so the `create()` tx succeeded but `TWAPOrderMathLib.calculateValidTo` then computed `part = (block.timestamp - 0) / t = ~3.3M`, which is far above the configured `n = 2` and triggers `AFTER_TWAP_FINISHED` on every `getTradeableOrderWithSignature` poll. Surfaced in the COW-1064 dry run (2026-06-18 report §6.4): supervisor logged the `0xc8fc2725...after twap finished` revert per block. Fix: - New `scripts/_twap_calldata.py` encodes the calldata fresh on every invocation with `t0 = int(time.time()) - 60` (backdated 60s so part 0 is Ready as soon as the order is on-chain). Module docstring explicitly warns against re-hardcoding t0. - `scripts/e2e-onchain.sh` Action 1 now shells out to the helper rather than carrying the hex inline. Validates the output is hex-shaped before passing to `cast send`. - `docs/operations/e2e-cow-1064-prep.md` section 2.3 step 3 replaces the pinned blob with a `python3 scripts/_twap_calldata.py` recipe and a historical note pointing at COW-1077. - `docs/operations/e2e-cow-1064-prep.md` section 4.2 recipe gets `import time` + `int(time.time()) - 60` for `t0` so the re-derivation flow does not re-introduce the bug. - `scripts/README.md` Action 1 description updated to mention the helper. Constants in the helper (sell/buy tokens, amounts, n, t, salt) mirror the prep doc's section 4.2; both must change in lockstep if the TWAP shape is retargeted. Validation: `python3 scripts/_twap_calldata.py` produces 516-byte calldata (1034 hex chars) starting with the correct selector `0x6bfae1ca`; the t0 word reflects current epoch (verified against `0x00...006a3537b5` on the smoke run). `bash -n scripts/e2e-onchain.sh` passes. No engine-side changes; this is a script-and-docs PR. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context). The original `t0 = 0` calldata was itself drafted by an earlier AI session; this PR returns the encoding to a hand-validated form.
mfw78 review of PR #8 (nullislabs#8) flagged "we already pull alloy, so pulling hex via there is really not much of a deal". The PR #47 (COW-1074) commit acc9654 then introduced two new custom hex helpers that recreate the same antipattern at a different scope: - `crates/shepherd-sdk/src/cow/app_data.rs::encode_hex` - 32-byte hash → `0x...`. Used by `resolve_app_data` to format the orderbook lookup path. - `modules/twap-monitor/src/strategy.rs::hex_short` - 8-byte prefix → `0x...…`. Used to format `appData` hashes in INFO log lines. Both crates already depend on `alloy-primitives` (sdk: 1.6, twap-monitor: 1.5), so the swap is a one-liner per call site: - `encode_hex(b)` → `format!("0x{}", alloy_primitives::hex::encode(b))` - `hex_short(b)` → `format!("0x{}…", alloy_primitives::hex::encode(&b[..8]))` Both functions keep their old signature so callers (`resolve_app_data` in the SDK, every `host.log` line in twap-monitor strategy) need no changes. Comments on both helpers now explicitly reference mfw78's PR #8 guidance so the next person tempted to hand-roll a `0123456789abcdef` table has a hook. Validation: cargo test -p shepherd-sdk -p twap-monitor: 32 + 23 passed; cargo clippy --all-targets -- -D warnings: clean; cargo fmt --check: clean; zero em-dash drift. Why this PR sits in a separate branch rather than amending PR #47: PR #47 is already In Review, and #48/#49/#50 stack on top of it. Amending would require force-pushing 4 branches. A small follow-up PR keeps each one bisectable and lets mfw78 review the alloy alignment in isolation. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context) while sweeping all open PRs for the same antipatterns mfw78 flagged on PRs #8 and #9.
Synthetic load test for shepherd's M4 stack. Distinct from: - COW-1064 (real Sepolia E2E, correctness, 90 min, 5 modules) - COW-1078 (backtest of 7d historical events, replay) - COW-1031 (7-day soak, wall-clock stability) This issue answers one question the others do not: how many events per block can the supervisor dispatch before something breaks? lgahdl's PR #9 review thread flagged sequential per-module dispatch as a potential bottleneck; this PR is how we measure it. Components added: 1. `tools/orderbook-mock` (new crate, axum-based) - HTTP server serving the two endpoints shepherd's cow-api host hits per submission. POST /api/v1/orders returns a synthetic 56-byte OrderUid; GET /api/v1/app_data/{hash} returns the empty appData document. CLI knobs: --port, --latency-ms, --error-rate (alternates InsufficientFee / InvalidSignature to exercise both TryNextBlock and Drop paths). 3 unit tests covering the happy path, the empty appData path, and the error-rate envelope. 2. `tools/load-gen` (new crate, alloy-based) - connects to Anvil, impersonates the pinned Sepolia test EOA via anvil_impersonateAccount + anvil_setBalance, then on every new block fires N ComposableCoW.create(...) + M CoWSwapEthFlow.createOrder(...) calls. Each create uses a fresh salt counter so submissions do not collide on the dedup check. 3 unit tests covering pinned address parsing, salt uniqueness, and calldata selector shape. 3. Engine config: ChainConfig gains optional `orderbook_url` (per chain). OrderBookPool::from_config honours the override using cowprotocol::OrderBookApi::new_with_base_url; absent overrides fall back to canonical api.cow.fi URLs. main.rs switches from ::default() to ::from_config(&engine_cfg). Useful long-term for staging/barn targets, immediately needed to point at the mock. 4. `engine.load.toml` - chain 11155111 -> ws://localhost:8545, cow base URL -> http://localhost:9999, metrics on 127.0.0.1:9100, state_dir = ./data/load (wiped per run). 5. Scripts: - `scripts/load-bootstrap.sh` brings up Anvil + orderbook-mock, tracks PIDs in /tmp/shepherd-load.pids, exposes a teardown helper. - `scripts/load-teardown.sh` idempotent cleanup. - `scripts/load-run.sh` orchestrates one scenario end-to-end: bootstrap, build modules, start engine, snapshot /metrics, run load-gen for --duration-min, snapshot /metrics again, tear down, drop a report skeleton at docs/operations/load-reports/load-NxM-YYYY-MM-DD.md. 6. `docs/operations/load-testnet-runbook.md` - operator runbook covering the three scenarios (baseline 5x5, medium 20x20, saturation 50x50), expected acceptance bars, what the test does NOT prove (WS reconnect / drift / real-orderbook fidelity), troubleshooting. Validation: - cargo test --workspace --exclude <wasm-only-modules>: 196 passed. - cargo clippy --workspace --all-targets --tests -- -D warnings: clean. - cargo fmt --all --check: clean. - bash -n scripts/load-{bootstrap,run,teardown}.sh: clean. - Live orderbook-mock smoke: POST returns valid 56-byte hex UID, GET returns {"fullAppData":"{}"}, /_stats reflects counters. Pending (not in this PR): - Baseline 5x5 report against a real Anvil fork - requires Bruno's RPC_URL_SEPOLIA_HTTP from scripts/.env; once that runs, the report lands in docs/operations/load-reports/. - Metrics-delta auto-generation in scripts/load-run.sh (left as TBD in the script; e2e-report-gen.sh has the delta logic we can adapt). - Saturation scenario - run after the baseline lands so the bottleneck has a clean baseline to compare against. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
…tion (COW-1079)
First COW-1079 run on a real Anvil fork of Sepolia. The engine-side
acceptance bar is cleared with wide margin:
- Per-block dispatch latency p50/p95/p99 = 4/6/7 ms (bar was < 2 s).
- Zero traps, zero poisoned modules, zero shepherd_module_errors_total.
- EthFlow strategy submitted 1 OrderPlacement end-to-end through the
mock orderbook in 10 ms; submitted:{uid} marker written cleanly.
- 63 Anvil blocks dispatched flawlessly.
The honest finding: load-gen's transactions get into Anvil's mempool
(twap_ok=270, ethflow_ok=270 per the eth_sendTransaction response),
but only 5 ConditionalOrderCreated + 1 OrderPlacement events
actually fired - the rest reverted at the contract level
(ComposableCoW.create + EthFlow.createOrder run preconditions the
load-gen-crafted bodies don't pass).
So this run stressed the engine with ~6 events over 60 s, not
5+5 per block. The bar criterion that depends on the load-gen
(events-per-block delivered) is the only one that doesn't pass;
filing a follow-up to calibrate the revert rate before re-running.
Report at docs/operations/load-reports/load-5x5-2026-06-19.md
mirrors the COW-1064 e2e-report shape and signs off as
"conditional pass" - engine meets the bar; load-gen needs work.
AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
scripts/lib.sh exports REPORTS_DIR=e2e-reports/ unconditionally. load-run.sh used to set REPORTS_DIR=load-reports/ BEFORE sourcing load-bootstrap.sh (which transitively sources lib.sh), so the override was lost and the auto-generated skeleton ended up under e2e-reports/ next to the COW-1064 reports. Move the assignment after the source so the load-reports/ path wins, with a comment explaining the ordering trap. Drive-by: removed the misplaced e2e-reports/load-5x5-2026-06-19.md from the first run; the committed report at load-reports/load-5x5-2026-06-19.md (commit 59fe714) is the canonical copy. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
COW-1079 baseline's 5/270 + 1/270 revert rate had two distinct root causes, both contract-side, neither shepherd's fault: 1. **Nonce race in burst submissions.** Anvil's `eth_sendTransaction` against an impersonated account auto-assigns a nonce when none is provided, but the assignment racts with the caller's burst submission. When load-gen fired 5 TWAP + 5 EthFlow per block without waiting for individual receipts, most txs landed in the mempool sharing the same nonce, and Anvil's miner included only one per block - the rest reverted as nonce-too-low. Fix: read the EOA's current nonce at boot, increment locally per successful submission, pin `tx.nonce` explicitly on every `TransactionRequest`. Lock-step with cargo build cache so the nonce counter never crosses async-boundary corruption. 2. **EthFlow OrderUid dedup on identical GPv2 OrderData.** The CoWSwapEthFlow contract dedups by the GPv2 `OrderUid` which is keccak over (buyToken, receiver, sellAmount, buyAmount, appData, feeAmount, validTo, partiallyFillable, kind, sellTokenSource, buyTokenDestination). quoteId is NOT part of that hash. The prior load-gen varied only `quoteId` per call, so all 270 EthFlow submissions produced the same UID and the contract rejected 269/270 as `OrderIsAlreadyOwned`. Fix: vary `sellAmount` by 1 wei per call (`BASE_SELL_AMOUNT + seq`) and pass that same value as `msg.value` so the contract's `msg.value == order.sellAmount` invariant holds. Re-ran baseline 5x5 after both fixes: 130/130 TWAP + 130/130 EthFlow delivered, 130 ConditionalOrderCreated + 130 OrderPlacement events on-chain, 130 cow_api submits OK to mock, 130 ethflow markers written, zero shepherd_module_errors_total. Updated baseline report at docs/operations/load-reports/load-5x5-2026-06-19.md from 'conditional pass' to 'full PASS' with the post-calibration numbers (TWAP block p99 = 49 ms, EthFlow log p99 = 11 ms, 40x margin on the < 2 s bar). Medium 20x20 and saturation 50x50 are now unblocked per the COW-1079 acceptance roadmap. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
…(COW-1079) Closes the COW-1079 three-scenario sweep with the COW-1080 calibration in place. All three scenarios pass: baseline 5x5 - 130/130 each, TWAP block p99=49ms medium 20x20 - 280/280 each, TWAP block p99=67ms saturation 50x50 - 300/300 each, TWAP block p99=78ms Latency growth across the watch-count range (130 -> 280 -> 300) is sub-linear: 49 -> 67 -> 78 ms. The lgahdl PR #9 concern about sequential per-module dispatch saturating under load is NOT surfaced at this scale. Zero shepherd_module_errors_total, zero traps, zero EthFlow submit errors across all three runs. The unexpected finding from saturation: the engine did not saturate. The bottleneck is load-gen's sequential eth_sendTransaction submission (each tx ~200 ms RTT, so 100 tx/iteration = ~20 s, vs. Anvil's 1 s block time). To genuinely saturate the engine we would need parallel load-gens against different impersonated EOAs, a sub-second block-time, or thousands of pre-seeded watches. EthFlow log p99 stayed flat at ~9 ms across all three scenarios (it is dominated by the cow-api submit roundtrip, not engine state), confirming the submit path scales independently of the watch count. The cold-start outlier (~500 ms on the first watch-heavy block) appears consistently across runs and is independent of the steady- state watch count - it is a one-shot first-block redb/eth_call warmup cost, NOT a saturation symptom. What this proves: - Shepherd M4 supervisor handles >= 300 concurrent watches + >= 138 block dispatch cycles in 2 min with p99 < 80 ms. - cow-api submit path is steady at ~9 ms p99 regardless of watch count. - Zero error/trap/poison across all three scenarios. What it does NOT prove (and is not in scope here): - Behaviour at 3000+ watches. - WS reconnect resilience (COW-1031 soak). - Multi-day memory drift (COW-1031). - Real-orderbook 4xx variety (COW-1078 backtest). COW-1079 ready to move to In Review. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
…079) The single-EOA saturation 50x50 report identified the per-EOA nonce serialisation as the bottleneck before the engine had a chance to saturate. This commit removes that bottleneck: load-gen: - New --parallel N flag. Each worker impersonates a synthetic EOA (0x57...01..0a), gets its own WS connection + nonce stream, runs its own per-block submission loop. Total events per block scales linearly with N. - Disjoint salt space per worker via 96-bit prefix. - Disjoint EthFlow sellAmount space via a 10_000-wide per-worker window (the first attempt shifted by 96 bits, blowing past the 1M ETH funded balance with 7.9e28 wei sellAmounts; fixed). scripts/load-bootstrap.sh + scripts/load-run.sh: - Accept --block-time (passes to anvil) and --parallel (passes to load-gen). Defaults preserve historic behaviour: --block-time 1, --parallel 1. - Auto-report filename now includes scenario label (load-NxM-SCENARIO-date.md) so saturation-parallel does not overwrite the baseline 5x5 report. Saturation-parallel run (10 workers x 5 TWAP + 5 EthFlow per block, --block-time 0.5, 2 min): - load-gen: 895/895 TWAP + 895/895 EthFlow acks, 0 errors. - engine saw 381 ConditionalOrderCreated + 343 OrderPlacement events (43% / 38% delivery vs load-gen acks - Anvil + WS dropping under the heavier load). - shepherd_module_errors_total = 0, zero traps. - All 343 EthFlow submissions reached the mock orderbook 1:1. - TWAP block dispatch: histogram p50/p99 = 145 ms, max = 101 593 ms (101 s outlier on one block when 380+ watches polled against a stressed Anvil JSON-RPC). - Engine-log dispatch_block: n=586, p50=4ms, p95=46ms, p99=74ms, max=101 593 ms - same outlier. Saturation knee identified: 380+ active watches + 0.5s block-time + 10 concurrent WS subscribers produces a 101-second worst-case dispatch + 38-43% event delivery loss. Both symptoms point at the surrounding system (Anvil + WS transport), not at shepherd; engine continues to scale sub-linearly with watch count and never produces a module error, trap, or panic under any tested configuration. For the 7-day COW-1031 soak: this implies the operator should use a paid Sepolia archive endpoint (Alchemy / drpc / QuickNode), not publicnode, OR accept event drops and rely on supervisor reconnect + eth_getLogs re-indexing. Documented in the new report. Report at docs/operations/load-reports/load-50x50-parallel-2026-06-19.md. AI assistance disclosure: drafted by Claude (Opus 4.7, 1M context).
…W-1084) Adds `tools/baseline-latency/baseline_latency.py`, a per-chain script that pairs every on-chain `EthFlow.OrderPlacement` event in a trailing window with the orderbook's record for the same UID and reports `(creationDate - block.timestamp)`. Matching is rigorous: the script ABI-decodes the GPv2OrderData from each event, computes the EIP-712 order digest against the chain's GPv2Settlement domain, and looks up the resulting UID against the orderbook's bulk `/account/.../orders` fetch (single-UID fallback if missed). No temporal-FIFO approximation. ## Headline finding For EthFlow orders the orderbook indexer sets `creationDate := block.timestamp` (not the indexer's ingest time), so the historical delta is structurally 0s on every chain. This is intentional back-fill-style behaviour, not a measurement bug. **Implication**: EthFlow indexer latency cannot be derived from historical orderbook data — the meaningful relayer-latency baseline lives on the TWAP lane (where the orderbook records the indexer's `now()` per child order PUT). TWAP child-latency is a follow-up; it needs per-part UID derivation from each parent `ConditionalOrderCreated` static input. Sepolia ran clean: 256 events scanned, 200 UID-derived pairs, all 200 matched against the bulk fetch (`bulk_hit=200`). Median = p95 = 0.0s, exactly as the finding predicts. ## Mainnet/Gnosis/Arbitrum/Base = RPC-LIMITED Public-tier RPCs (drpc.org free, 1rpc.io, ankr w/o key, llamarpc, cloudflare-eth) all refuse / throttle `eth_getLogs` at any usable chunk size on the production chains. The script halves down to 50-block chunks and gives up after 3 consecutive failures, marking the cell `RPC-LIMITED` with a pointer to the `RPC_URL_*` env override. This is the same constraint the M5 soak (COW-1031) will face and independently confirms the paid-endpoint requirement for any serious log-scanning workload. ## Files - `tools/baseline-latency/baseline_latency.py` (~520 lines): argparse CLI, per-chain `Chain` dataclass, JSON-RPC helper with halving retry + `RpcLimited` sentinel, EIP-712 order digest + UID derivation, UID-keyed orderbook matching, markdown report renderer. - `tools/baseline-latency/data/*.json`: per-chain raw dump (events, pairs, deltas, diagnostics) for auditability. - `docs/operations/baselines/baseline-latency-2026-06-19.md`: the first run's report. ## Why this matters for the grant Pinning the orderbook's `creationDate` semantics matters because the COW-1079 and COW-1031 KPIs reference "watchtower latency" — the M4 report needs to be honest about which lane the latency lives on (TWAP relayer PUT, not EthFlow indexer ingest). The Sepolia data set also gives the M4 e2e harness ground-truth UID ↔ block pairings to cross-check against. AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end and validated against live Sepolia data by the operator.
…W-1082) The chain backend previously dropped alloy's structured `RpcError::ErrorResp` payload on the floor — the formatted error string went into `HostError.message`, but `HostError.data` stayed `None` and `HostError.code` was hard-coded to `-32603`. That made the twap-monitor's poll-time revert classifier inert on real traffic: `OrderNotValid` / `PollNever` / `PollTryAtBlock` / `PollTryAtEpoch` all fell through to `TryNextBlock` because `decode_revert_hex` only fires on a non-empty `err.data`. This change wires the structured payload through end-to-end. ## Changes - `crates/nexum-engine/src/host/provider_pool.rs`: when alloy's `provider.raw_request` fails with an `RpcError::ErrorResp`, the pool now captures both `payload.code` (as `Option<i64>` so we can distinguish "no ErrorResp" from "ErrorResp with code 0") and `payload.data` (as `Option<String>`, the JSON-encoded revert hex) and surfaces them on `ProviderError::Rpc`. Transport-side failures (timeout, websocket disconnect) leave both `None`. The two subscribe paths (`subscribe_blocks`, `subscribe_logs`) keep `code: None, data: None` since they don't carry an ErrorResp. - `crates/nexum-engine/src/host/impls/chain.rs`: extract the `ProviderError -> HostError` projection into a free helper `provider_error_to_host_error`. The `Rpc` arm forwards the structured `data` verbatim, preserves the node-reported code (saturating out-of-`i32` values to `-32603`), and falls back to `-32603` only when no `ErrorResp` was present. Five unit tests cover: revert with data, transport failure with `None`, out-of-range code, unknown-chain, and invalid-params. - `modules/twap-monitor/src/strategy.rs`: update the stale comment on the `decode_revert_hex` branch — that branch is now live on real traffic, the only `None` path is transport-level failures (which keep the safe `TryNextBlock` default). ## Why this is M4, not correctness No incorrect order is ever submitted (the contract reverts; the orderbook never sees a bad body). The issue is pruning efficiency: a permanently dead TWAP watch was re-polled every block until a submit eventually failed for an unrelated reason, and the local-store filled with `watch:` entries the strategy could otherwise drop on the first revert. With this fix the SDK-side classifier dispatches `Drop` / gate on the first revert, matching the documented expectation in `docs/adr/0007-upstream-protocol-logic-to-cow-rs.md`. ## Tests - 70/70 nexum-engine tests pass - 23/23 twap-monitor tests pass - 5/5 new chain.rs projection tests pass (revert-with-data, transport-fail, out-of-range-code, unknown-chain, invalid-params) - `cargo clippy -p nexum-engine -p twap-monitor --all-targets -- -D warnings` clean ## Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on `modules/twap-monitor/src/strategy.rs:189`. The mirror of this fix on the cow-api side is COW-1075 (already merged via PR #48). AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end and validated against the existing twap-monitor strategy tests before push.
…OW-1083)
The strategy's `apply_submit_retry` previously wrote an empty
`backoff:{uid}` marker on every retriable submit failure (including
the `TryNextBlock` fallback for unparseable orderbook envelopes). The
marker was a presence flag with no payload, so on every supervisor
reconnect / engine restart the same dead placement would retry
indefinitely — bounded only by log re-delivery frequency.
This change persists a per-UID retry count in the marker's value
(ASCII `u32`) and upgrades to `dropped:` after `MAX_BACKOFF_RETRIES =
5` consecutive retries. The upgrade emits a Warn-level log line so
the operator sees the structural issue (flaky CDN, indexer hiccup,
poisoned envelope) rather than silently accumulating retries.
## Changes
- `modules/ethflow-watcher/src/strategy.rs`:
- New const `MAX_BACKOFF_RETRIES = 5`.
- New helper `read_backoff_count` that reads + parses the marker
payload; pre-COW-1083 empty markers decode to 0 so previously-set
backoff: rows still get a fresh attempt (no premature drop on
rollout).
- `apply_submit_retry`'s retriable branch now reads the prior
count, increments, and either writes the new count or upgrades
to `dropped:` (clearing the stale `backoff:`) at the cap.
- Cap-upgrade log line carries the retry-count and message: "...
after 5 retries on transient/unparseable rejection ...".
## Tests
- 19/19 ethflow-watcher tests pass.
- New `submit_transient_error_at_cap_upgrades_to_dropped_warn`:
seeds `backoff:{uid} = "4"`, triggers a `data: None` rejection
(the unparseable case the issue names explicitly), asserts:
* `dropped:{uid}` is now set
* `backoff:{uid}` is cleared (single outcome marker at rest)
* exactly one Warn log line containing "ethflow dropped" +
"retries"
- New `submit_transient_error_with_legacy_empty_marker_resets_counter`:
backwards-compat — a pre-COW-1083 empty `b""` marker is treated
as count 0, bumped to "1" on first retry rather than prematurely
dropping. Protects in-flight backoffs across the rollout.
- Existing `submit_transient_error_writes_backoff_marker_and_returns`
extended with an assertion that the first retry persists
`backoff:{uid} = "1"`.
- `cargo clippy -p ethflow-watcher --all-targets -- -D warnings`
clean.
## Why this is M4
Surfaced by jeffersonBastos's PR #55 (M3 mirror) review, thread on
`crates/shepherd-sdk/src/cow/error.rs:82`. Latent in normal
operation (the host forwards parseable envelopes after COW-1075, so
`classify_api_error` returns `Drop` for permanent rejections), but
the gap fires when the orderbook returns a non-JSON 4xx body
(e.g. an HTML error page from a CDN) or if a future host change
accidentally drops the envelope again. Bounded retry semantics
close the latent risk without changing the safe-default
classification (still `TryNextBlock` on `None` data — that part is
explicitly out of scope per the issue).
AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end
and validated against the existing ethflow-watcher strategy tests
before push.
Adds the COW-1078 pre-soak backtest end-to-end:
1. `tools/backtest-collect/backtest_collect.py` — Python collector
that pulls a trailing N-day window of `OrderPlacement` (EthFlow)
and `ConditionalOrderCreated` (TWAP) events from Sepolia,
ABI-decodes each payload, derives the EthFlow `OrderUid` via
EIP-712 against the chain's GPv2Settlement domain, resolves every
non-empty `appData` hash via `GET /api/v1/app_data/{hash}`, and
emits a single fixtures JSON. Reuses the log-scan + UID-derive
infra introduced by the baseline-latency tool (COW-1084 PR #57).
2. `crates/shepherd-backtest` — new Rust binary that loads the
fixtures, programs a `MockHost` per event (resolved `app_data`
response + UID-echo submit response), and drives
`ethflow_watcher::strategy::on_logs` directly. Each event is
classified into `Submitted` / `RejectedExpected` /
`RejectedUnexpected` / `StrategyError` and rendered into a
Markdown report at `docs/operations/backtest-reports/
backtest-7d-YYYY-MM-DD.md`.
3. `modules/ethflow-watcher` — `crate-type = ["cdylib", "rlib"]`
and cfg-gate the wit-bindgen glue so the rlib carries only the
`strategy` module (now `pub mod`) for native consumers. The
wasm artefact is unchanged.
## First run
7-day Sepolia window (2026-06-15..2026-06-22): **240 EthFlow events,
240 Submitted, 0 anomalies = 100.0% pass vs. 95% threshold**. The
report is committed at
`docs/operations/backtest-reports/backtest-7d-2026-06-22.md`.
26 TWAP `ConditionalOrderCreated` events are collected and counted
but the replay is deferred to Phase 2B — driving
`twap_monitor::strategy::on_block` requires walking each watch's
`eth_call(getTradeableOrderWithSignature)` per-block, which
public-tier RPCs refuse (see the baseline-latency / COW-1031
finding). The fixtures are committed so the future re-run inherits
the same dataset.
## Scope
- v1: EthFlow lane end-to-end (collector + replay + report).
- v2 (follow-up): TWAP lane via paid-RPC archive walking; downstream
validation via `POST /api/v1/quote` round-trip on captured
bodies.
- Out of scope per the issue: supervisor / event-loop / WS reconnect
coverage (stays on the wall-clock soak); fuel/memory limits (stays
on COW-1036 / soak); orderbook PUT mutation (forbidden — only
read-only endpoints are touched).
## Tests
- 19/19 ethflow-watcher tests pass (rlib + cdylib build both clean)
- Full workspace test sweep passes (no regressions)
- `cargo clippy -p shepherd-backtest -p ethflow-watcher --all-targets
-- -D warnings` clean
- Live run: 240 fixtures → 240 Submitted, 0 anomalies
## Reproducing
```bash
python3 tools/backtest-collect/backtest_collect.py --days 7
cargo run -p shepherd-backtest -- \
--fixtures tools/backtest-collect/fixtures-YYYY-MM-DD.json
```
AI-assisted authoring with Claude (Opus 4.7); reviewed end-to-end
and validated against a live Sepolia 7d window before push.
Closes the M5 packaging gap surfaced by the audit: the Dockerfile +
compose recipe lived inside `docs/production.md` but neither was at
the repo root, so `docker build .` didn't work and there was no
published image. This change makes the deploy path one-line on a
fresh VM.
## What ships
- **`Dockerfile`** — multi-stage build (rust:1.96-slim-bookworm →
debian:bookworm-slim). Builds the engine in release + the 5
production modules to wasm32-wasip2. Runtime stage strips down to
`tini` (PID 1 for graceful shutdown / SIGINT forwarding per
COW-1072) + `ca-certificates` (TLS to cow.fi + paid RPCs) + a
non-root `shepherd` user owning `/var/lib/shepherd`. Final image:
**198 MB** (engine + 5 wasm modules + Debian slim).
- **`.dockerignore`** — excludes `target/`, `data/`, the heavy
backtest / baseline JSON fixtures, and local-only engine configs,
while keeping `modules/fixtures/*-bomb` (workspace members; Cargo
rejects the manifest if they're missing) and the source markdown
docs (so `docker exec` can grep them in place).
- **`docker-compose.yml`** — two profiles. Default boots just the
engine with a `shepherd-state` named volume + the operator's
`./engine.toml` mounted ro at `/etc/shepherd/engine.toml`, metrics
on the host loopback (`127.0.0.1:9100`). The `observability`
profile (`docker compose --profile observability up`) layers a
Prometheus container pre-wired to scrape `shepherd:9100`. Graceful
shutdown via `stop_signal: SIGINT` + `stop_grace_period: 30s` per
the production runbook. Healthcheck hits `/metrics`.
- **`engine.docker.toml`** — pre-baked config that matches the
paths the image bakes (`/opt/shepherd/modules/*.wasm`,
`/opt/shepherd/manifests/*.toml`, `/var/lib/shepherd` state
dir). Operator workflow: `cp engine.docker.toml engine.toml`,
swap `<RPC_KEY>` placeholders, `docker compose up -d`.
- **`docs/deployment/docker.md`** — operator runbook. Covers
first-boot, engine.toml configuration, upgrade / rollback,
local-build path, post-deploy verification, cross-links to
`docs/production.md` for the full hardening surface.
- **`docs/deployment/prometheus.yml`** — scrape config consumed by
the observability compose profile.
- **`.github/workflows/docker.yml`** — build + push to
`ghcr.io/bleu/nullis-shepherd` on every push to `main` and every
`v*` tag. PR builds run the build for smoke (no push). Tags
produced: `latest` (main HEAD), `v<tag>` (releases),
`sha-<short>` (every event for exact pinning), `manual-<run_id>`
(workflow_dispatch). Registry-side layer cache via
`:buildcache` keeps incremental rebuilds fast. linux/amd64 only —
the soak VM is x86_64; add arm64 once an operator surfaces a
real need. Action SHAs pinned to match `.github/workflows/ci.yml`
style.
## Smoke validation
Build runs locally end-to-end in ~10 min on a clean Docker daemon:
$ docker build -t shepherd:smoke .
$ docker run --rm shepherd:smoke --help
usage: nexum-engine [<wasm-path> [<manifest-path>]] \
[--engine-config <path>] [--pretty-logs]
$ docker run --rm -v "$PWD/engine.docker.toml:/etc/shepherd/engine.toml:ro" \
shepherd:smoke
{"level":"INFO","message":"nexum-engine starting",...}
{"level":"INFO","message":"metrics exporter listening at /metrics",...}
{"level":"INFO","message":"opening chain RPC provider","chain_id":1,...}
Error: connect chain 1: HTTP format error: invalid uri character
^- expected: <RPC_KEY> placeholder not a real URL
Proves: image builds, entrypoint forwards CMD, engine loads
`/etc/shepherd/engine.toml`, metrics exporter binds, provider pool
iterates the configured chains, graceful error path works.
## Tests
- [x] Local `docker build .` succeeds (rust:1.96 base — wasmtime 45
requires rustc >= 1.93, the docs/production.md `1.86` pin was
stale)
- [x] Image size: 198 MB
- [x] `docker run ... --help` works
- [x] `docker run ... -v engine.docker.toml:...` reads config + binds
metrics + iterates chains
- [x] `cargo test --workspace` clean (18 groups, 203 passed, 0 failed)
## Reproducing the soak deploy
On a fresh Debian/Ubuntu VM with Docker installed:
```bash
git clone https://github.com/bleu/nullis-shepherd /opt/shepherd
cd /opt/shepherd
cp engine.docker.toml engine.toml
$EDITOR engine.toml # add real RPC URL
docker compose pull # once ghcr.io image is published
docker compose up -d
docker compose logs -f shepherd
curl -s http://127.0.0.1:9100/metrics | head -50
```
## Follow-ups for M5 (separate PRs)
- `docs/deployment/multi-chain-guide.md` — dedicated walkthrough
configuring 4 chains together (Mainnet + Gnosis + Arbitrum + Base)
with per-chain module subscriptions
- Example module declaring multi-chain support (every current
example pins Sepolia)
- Optional automated CD trigger (workflow_dispatch SSH'ing to the
soak VM to pull + restart) — gated on SSH_PRIVATE_KEY repo secret
AI-assisted authoring with Claude (Opus 4.7); smoke-validated end-
to-end via a local docker build + run before push.
Companion to the M5 Docker packaging — the operator workflow is `cp engine.docker.toml engine.toml` then drop in a paid RPC URL. Without this rule a clumsy `git add -A` could commit the key. The committed sibling templates (engine.example/docker/m2/m3/e2e/load.toml) stay trackable. Validated against a live smoke run: drpc Sepolia WSS endpoint pasted into engine.toml, `docker compose up`, engine subscribed to newHeads + logs, 6 sequential blocks dispatched (11117171..76), metrics `shepherd_event_latency_seconds` p99 = 0.14ms. Tear-down clean. No engine.toml ever staged.
Closes the footgun surfaced by the M5 smoke run on drpc Sepolia:
configuring `rpc_url = "https://..."` for a chain that the modules
subscribe to silently degrades to an infinite WARN-with-backoff loop
(COW-1071's reconnect retries forever because `eth_subscribe` is
WS-only in the JSON-RPC spec). Three coordinated changes:
## 1. Boot-time validation
`EngineConfig::validate_transports()` walks every `[chains.<id>]`
entry, and for any `rpc_url` not starting with `ws://` / `wss://`
emits one loud ERROR-level structured log line with:
- the chain id
- the redacted offending URL
- the redacted suggested `wss://` swap
- actionable copy explaining the WS requirement and the escape
hatch (`[chains.<id>] require_ws = false` for poll-only chains
that never subscribe)
The validator is invoked from `main.rs` AFTER the tracing
subscriber is initialised (calling it inside `load_or_default`
silently dropped the log).
A `require_ws: bool` field is added to `ChainConfig` with
`#[serde(default = "default_require_ws")]` = `true`. Operators who
genuinely need an HTTP endpoint (poll-only modules, no block / log
subscriptions on this chain) opt out explicitly per chain.
## 2. URL redaction in boot logs
The pre-existing `opening chain RPC provider` log in
`provider_pool::from_config` was emitting the full URL — API key
included — at INFO level. Log aggregators (Loki / Datadog / Splunk)
routinely retain weeks of these lines; the key has no business
sitting in cold storage. The new `engine_config::redact_url` helper
(public so other call sites can adopt it) replaces any path segment
longer than 20 chars that doesn't contain `.` or `:` with `<KEY>`.
Matches Alchemy / drpc / Infura / QuickNode key shapes.
Same helper is used for both the validation ERROR's `rpc_url` and
`suggested` fields and the provider-pool boot log.
## 3. Docs + example cleanup
- `engine.example.toml`: every chain entry switched to `wss://`,
with a header block explaining the WS requirement + the
`require_ws = false` escape hatch. The previous mix of `https://`
+ `wss://` would have tripped the new validator on its own example.
- `docs/production.md §6`: blockquote callout pointing operators at
the WS requirement, redaction behaviour, and the escape hatch.
## Validation evidence
Smoke 1 (HTTP, expected to ERROR):
{"level":"ERROR","message":"rpc_url uses HTTP transport but the engine subscribes to blocks/logs via eth_subscribe (WS-only). [...]","chain_id":11155111,"rpc_url":"https://lb.drpc.live/sepolia/<KEY>","suggested":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
Smoke 2 (WSS, expected to pass + redacted):
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,"url":"wss://lb.drpc.live/sepolia/<KEY>",...}
$ grep -c "<the-actual-key>" smoke.log
0
## Tests
- 9 new unit tests in `engine_config::tests`:
* `validate_accepts_wss_url`, `validate_accepts_ws_url`
* `validate_is_silent_when_require_ws_is_false`
* `validate_runs_without_panicking_on_http_url`
* `suggest_swaps_https_to_wss`, `suggest_swaps_http_to_ws`,
`suggest_passes_through_already_ws_url`
* `redact_replaces_long_path_segments`,
`redact_keeps_short_segments_intact`
- Workspace: 18 groups, **212 passed, 0 failed** (was 203 → +9)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
AI-assisted authoring with Claude (Opus 4.7); validated against the
live drpc Sepolia endpoint via two smoke runs (HTTP fail, WSS
pass) before push.
Operator workflow before this change forced the paid-RPC URL to
live in a file (`engine.toml`), which is fine for systemd but
awkward for Docker/compose: the URL had to be hand-edited inside a
volume-mounted file, secrets and config got tangled, and the
internal drpc test key was at risk of slipping into a committed
example. This change makes the engine treat `${VAR_NAME}` tokens
inside `engine.toml` as environment-variable references, resolved
at config-load time:
[chains.11155111]
rpc_url = "${SEPOLIA_RPC_URL}"
The `engine.docker.toml` and `engine.example.toml` templates ship
with `${VAR}` placeholders for all five chains, so the committed
files stay secret-free regardless of deployment path.
## Operator workflow (Docker)
cp .env.example .env
$EDITOR .env # paste real wss:// URLs
docker compose up -d
`docker compose` reads the repo-root `.env` automatically (already
the compose default) and forwards the named variables into the
container via the new `environment:` block; the engine substitutes
them when parsing `/etc/shepherd/engine.toml`.
## Implementation
- `engine_config.rs::substitute_env_vars` — hand-rolled parser
(no regex dep) that walks the raw TOML text, matches `${NAME}`
tokens against `[A-Z_][A-Z0-9_]*`, and looks each up via
`std::env::var`. Three error variants via `thiserror`:
* `Missing { name }` — variable referenced but unset; message
includes the exact name and a pointer to the `.env` workflow.
* `InvalidName { name }` — typo (lowercase, leading digit);
suggests the upper-cased variant.
* `Unclosed { offset }` — `${` without matching `}`.
- Called from `load_or_default` before `toml::from_str`, so the
substitution layer never sees parsed TOML — a missing env var
surfaces with the exact variable name, not a downstream
"invalid URI character" several layers deep.
- Substitution runs over the whole file (comments included; harmless).
## Companion changes
- `.env.example` — committed template with placeholders for all 5
chain `*_RPC_URL` variables + the optional `SHEPHERD_IMAGE` and
`SHEPHERD_ENGINE_CONFIG` overrides.
- `.gitignore` — adds `!.env.example` exception so the template
stays trackable while `.env` and `.env.local` etc. stay ignored.
- `docker-compose.yml` — passes the five `*_RPC_URL` env vars
through to the container; the engine config bind-mount now
defaults to `engine.docker.toml` (the committed template) and
honours `SHEPHERD_ENGINE_CONFIG` for operators who prefer a
bespoke file.
- `engine.docker.toml` + `engine.example.toml` — every `[chains.*]`
entry switched to `${*_RPC_URL}` placeholders. Header comments
spell out the workflow.
- `docs/deployment/docker.md` — first-boot section now leads with
`cp .env.example .env` (was `cp engine.example.toml engine.toml
&& edit`). §2 explains the bind-mount + the
`SHEPHERD_ENGINE_CONFIG` escape hatch.
## Validation
Smoke 1 (compose end-to-end):
$ cp .env.example .env
$ echo "SEPOLIA_RPC_URL=wss://lb.drpc.live/sepolia/<real-key>" >> .env
$ echo "SHEPHERD_ENGINE_CONFIG=./engine.local.toml" >> .env
$ docker compose up -d
...
{"level":"INFO","message":"opening chain RPC provider","chain_id":11155111,
"url":"wss://lb.drpc.live/sepolia/<KEY>",...} ← env-resolved, key redacted
{"level":"INFO","message":"supervisor up","loaded":2,"alive":2,...}
{"level":"INFO","message":"block subscription open","chain_id":11155111,...}
{"level":"INFO","message":"log subscription open","module":"twap-monitor",...}
{"level":"INFO","message":"log subscription open","module":"ethflow-watcher",...}
$ docker compose logs | grep -c <real-key>
0 ← zero leaks
$ curl -s http://127.0.0.1:9100/metrics | grep latency_seconds_count
shepherd_event_latency_seconds_count{module="twap-monitor",event_kind="block"} 4
Smoke 2 (missing env var, expected fail-fast):
$ unset SEPOLIA_RPC_URL
$ docker compose up
Error: engine config env-var substitution failed: environment variable
`SEPOLIA_RPC_URL` referenced via ${SEPOLIA_RPC_URL} in engine.toml but
not set. Export it before launching the engine (e.g. via a `.env`
file consumed by `docker compose`).
## Tests
- 7 new unit tests in `engine_config::tests`:
* `substitute_replaces_known_variable`
* `substitute_errors_on_missing_variable`
* `substitute_errors_on_invalid_name`
* `substitute_errors_on_unclosed_brace`
* `substitute_passes_text_with_no_placeholders_through`
* `substitute_handles_multiple_placeholders_in_one_line`
* `substitute_preserves_utf8_around_placeholder`
- Workspace: 18 groups, **219 passed, 0 failed** (was 212 → +7)
- `cargo clippy --workspace --all-targets -- -D warnings` clean
AI-assisted authoring with Claude (Opus 4.7); validated end-to-end
against drpc Sepolia via the documented `.env` -> compose -> engine
workflow before push (no key ever staged).
VM smoke surfaced a false-negative `(unhealthy)`: the compose healthcheck called `wget` but the runtime image is built on debian:bookworm-slim which doesn't include it (only ca-certificates + tini, intentionally minimal). `wget: not found` → exit 127 → unhealthy mark, despite the engine actually working (21 blocks dispatched in 3 min, p99 latency 0.09ms, zero errors). Swap to bash's `/dev/tcp` builtin (always present in bookworm-slim's `/bin/bash`). Successful TCP open on the metrics port proves the exporter bound, which only happens after the supervisor finishes boot — same semantic, no image growth.
First fix attempt swapped wget for `/dev/tcp` but kept `CMD-SHELL`, which routes through `/bin/sh` (dash on debian:bookworm-slim). dash doesn't have the `/dev/tcp/<host>/<port>` builtin — it's bash- only. Probes failed with "cannot create /dev/tcp/...: Directory nonexistent". Switch to `CMD ["bash", "-c", ...]` so the bash builtin actually resolves. `bash` ships in the slim base; verified via `docker exec shepherd which bash` → `/usr/bin/bash`.
Cherry-pick of PR #62 + PR #63's redesign onto the M5 host runtime (env-var substitution in engine.toml, healthcheck fixes, etc) for the Sepolia soak VM. The PR review continues on the proper layered branches: - PR #62 — M2/BLEU-833 layer observe design - PR #63 — M3/M4 BLEU-855 split + COW-1074 cow_api_request integration This branch is deploy-only: it lets the soak run on the redesigned ethflow-watcher with the latest host runtime while review iterates on the layered PRs. After merge, this branch can be deleted; CI will republish ghcr.io/bleu/nullis-shepherd:latest with the merged design and the VM rolls forward to the official image. See COW-1076 for the full empirical evidence.
…store (COW-1085)
`getTradeableOrderWithSignature` returns the same Ready tuple in every
poll-tick during a TWAP child's validity window — the on-chain
conditional order has no way to know shepherd already POSTed it. The
strategy already wrote a `submitted:{uid}` marker after a successful
submit, but the next poll-tick polled the chain and submitted again,
producing a wasted orderbook call and a misleading
`DuplicatedOrder` Warn line in every soak that runs a TWAP.
Live evidence (2026-06-23 Sepolia soak):
10:02:36.784 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:37.190 INFO submitted submitted:0xd7116bd2...
10:02:48.870 INFO poll watch:0x8fab71c0...:0x93b1626c... -> Ready
10:02:49.855 WARN submit dropped watch (400): orderbook error
(DuplicatedOrder): order already exists
The first submission succeeded (`GET /api/v1/orders/0xd7116bd2...`
returns `status: fulfilled`); the second was wasted work.
The fix: at the top of `submit_ready`, compute the client-side UID
deterministically from the on-chain `(order, owner, chain)` tuple via
`OrderData::uid` and check `submitted:{uid}` in local-store; skip the
submit (and the appData resolve that precedes it) when the marker
exists. The marker write site is also updated to use the
client-computed UID for the key so the read and write paths agree
(in production the server-returned UID is the same value — both sides
derive it from the canonical `digest || owner || valid_to` layout —
and a divergence is now surfaced via a Warn).
Tests (24, all green natively + wasm32-wasip2):
* Existing `poll_ready_submits_order_and_persists_submitted_uid` and
`poll_ready_resolves_non_empty_app_data_then_submits` updated to
compute the expected marker key via `compute_uid_hex` instead of
hardcoding `submitted:0xfeedface` (the mock orderbook's stub UID,
which now triggers the divergence Warn so we also assert that).
* New `poll_ready_skips_submit_when_submitted_uid_already_in_store`:
seeds the marker, dispatches a block tick, asserts
`submit_order` (and the preceding appData resolve) are NOT called
and that the expected Info log appears.
Out of scope (deferred): the same idempotency pattern could be
applied to ethflow-watcher's `observed:{uid}` marker (already correct
there — the GET-not-POST design makes this naturally idempotent).
…econnects (COW-1086)
Adds a positive-recovery Info log when the block subscription
resumes after a silence ≥ 60 s, covering the observability gap
identified in the 2026-06-23 Sepolia soak.
## Background
The 2026-06-23 soak surfaced this sequence:
09:05:43 ERROR WS connection error: WebSocket protocol error:
Connection reset without closing handshake
target=alloy_transport_ws::native
(no further WS-related lines for ~1 h)
10:02:24 INFO indexed watch:... ← twap-monitor activity resumes
10:05:24 INFO ethflow observed ... ← ethflow-watcher activity resumes
`docker ps` showed 0 restarts and the container stayed healthy
throughout — alloy's transport layer reconnected internally without
the engine's `reconnecting_block_task` ever observing
`inner.next().await -> None`. So the engine never entered its
"stream ended → backoff → subscription reopened" path, and the
existing `block subscription reopened` Info log (COW-1071) never
fired. The transport-layer ERROR followed by silence is
indistinguishable from a hung engine on a soak dashboard.
## What changes
In `reconnecting_block_task`, on every yielded item compare
`now.duration_since(last_event)` against `BLOCK_GAP_LOG_THRESHOLD`
(60 s, 5× Sepolia block time). When the gap meets or exceeds the
threshold, emit:
INFO chain_id=... gap_s=... kind="block"
"stream gap closed - first event after silence
(likely an alloy-internal transport reconnect)"
The gap-detection logic is factored into a small synchronous helper
`block_stream_gap_to_log(now, last_event, threshold) -> Option<Duration>`
so it can be unit-tested without spinning up an async runtime or a
real provider.
## Why blocks only (not logs)
Block subscriptions have predictable cadence — Sepolia produces a
new block every ~12 s, mainnet every ~12 s. A 60 s silence is
therefore anomalous and worth surfacing. Log subscriptions, by
contrast, are inherently sparse (driven by on-chain user activity),
so the same threshold would fire false positives on quiet windows.
The existing `log subscription reopened` log already handles the
engine-detectable reconnect for log streams.
## Tests
4 new unit tests on the gap-detection helper:
* `block_stream_gap_to_log_returns_none_when_no_prior_event`
* `block_stream_gap_to_log_returns_none_when_under_threshold`
* `block_stream_gap_to_log_returns_some_at_threshold_boundary`
* `block_stream_gap_to_log_returns_some_when_well_over_threshold`
All 90 nexum-engine tests pass (86 existing + 4 new). Clippy strict
clean, fmt clean. Wasm build untouched.
## Out of scope
* End-to-end test of `reconnecting_block_task` against a mock
provider — no existing scaffolding for that path, and the gap
helper covers the decision logic deterministically.
* Suppressing or downgrading the `alloy_transport_ws::native` ERROR
itself — it is a legitimate transport-layer event, just one whose
recovery wasn't previously observable. The new Info line closes
that loop without losing the original signal.
## Live validation
The next time alloy auto-reconnects internally on the soak VM, the
new line will surface as a structured JSON event with
`gap_s=<seconds>` so the soak dashboard can correlate it with the
preceding transport ERROR.
) Squash of PR #68 - 9 markdown files reconciled (5 vapor items rephrased as future direction + capability-gating diagrams aligned to link-time enforcement). Verified: cargo doc --workspace --no-deps clean.
Author
|
Superseded by epic/m5-delivery + new PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Delivers M5 milestone of the Shepherd grant: Docker image + ghcr CI + compose, pre-soak backtest harness, baseline-latency tool, chain-forward error data, backoff cap, runtime ops fixes (env-var substitution, healthcheck shell, RPC key redaction, twap skip-submitted-uid, log block-stream gap closures), rust-idiomatic compliance, and docs reconciliation across M2-M5.
Changes
Aggregates
Feature PRs #57, #58, #60, #61 + compliance PRs #66/#67/#68 + 4 superseded-by-different-commit PRs (#62-#65).
Closes COW-1078, COW-1082, COW-1083, COW-1084, COW-1085, COW-1086
Breaking changes
None - this is the cumulative milestone delivery; all individual breaking-change discussions happened in the now-closed feature PRs.
Testing
cargo fmt --allcleancargo clippy --all-targets -- -D warningscleancargo test --all-featurespassing across all milestone-touching cratesAI assistance disclosure
This epic PR was opened by Claude Code (Opus 4.7) to consolidate M5 work into a single mergeable surface, fixing the Linear ticket transition gap where the original feature PRs were closed (not merged) when
dev/m5-baseadvanced. A human (Bruno) is accountable for the result.