test(supervisor): multi-chain isolation regression tests (COW-1073)#43
Closed
brunota20 wants to merge 1 commit into
Closed
test(supervisor): multi-chain isolation regression tests (COW-1073)#43brunota20 wants to merge 1 commit into
brunota20 wants to merge 1 commit into
Conversation
The supervisor's dispatch path is per-chain by construction (`dispatch_block(block)` filters modules by `block.chain_id` matching their `[[subscription]]` table), and the COW-1071 WS reconnect tasks own one per-chain backoff timer each. Multi-chain isolation is therefore structural, not derived. This PR locks the guarantee into the test suite with two new integration tests + a supervisor.rs docstring stating the invariant explicitly. ## New tests `multi_chain_dispatch_isolates_modules_by_chain`: - Boot two `example` modules with different `[[subscription]]` chain_ids (1 + 100). - Dispatch a block on chain 1 -> only module-a receives it (dispatched=1, alive_count=2 unchanged). - Dispatch a block on chain 100 -> only module-b receives it. - Validates: subscription filter is per-chain; a block on one chain does not even enter modules subscribed to a different chain. `multi_chain_poisoned_module_does_not_affect_other_chains`: - Boot fuel-bomb (always-traps) on chain 1 + example (healthy) on chain 100, with `PoisonPolicy::new(2, 60s)`. - Trap bomb #1 on chain 1 -> bomb dies, poisoned=0, example untouched. - Dispatch on chain 100 -> example receives (1/1). - Wait 1.1 s (bomb backoff window), trap bomb #2 -> poisoned=1. - Dispatch on chain 100 again -> example STILL receives. - Validates: a permanently-poisoned module on one chain does not consume restart slots, fuel, or scheduling attention from modules on any other chain. Total wall-clock ~1.2 s for the second test (one backoff window). ## supervisor.rs docstring The module-level comment now articulates the multi-chain isolation invariant explicitly so a future reader of the dispatch path knows the property is load-bearing. ## What this proves Supervisor side (dispatch fast-path): - Per-module `alive`, `failure_count`, `next_attempt`, `poisoned` are independent of which chain triggered the event. - Subscription filter excludes mismatched modules before any dispatch / restart logic runs. Upstream side (already proven by COW-1071's architecture): - `open_block_streams` spawns one task per chain; tasks share no state. A chain-A WS drop changes only chain-A's task state. - `open_log_streams` is per-(module, chain) -> even tighter isolation than block streams. ## Out of scope - A unit test that "fakes a WS drop" on chain A while chain B keeps yielding. Requires mocking `ProviderPool::subscribe_blocks` which today goes through real alloy / tokio infrastructure. The COW-1064 (E2E 4-6h testnet) and COW-1031 (7-day soak) will exercise this path against live RPCs. - Per-chain configurable backoff / health-window. Today the reconnect policy is workspace-wide; per-chain tuning is a 0.3 follow-up. ## Workspace impact - `cargo test --workspace` -> 163 host tests + 6 doctests passing (was 161 + 6; +2 from the new integration tests). - `cargo clippy --all-targets --workspace -- -D warnings` clean. - `cargo fmt --all --check` clean. Linear: COW-1073. Ninth M4 issue landed; stacks on #42 (COW-1072).
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Locks the multi-chain isolation guarantee into the test suite.
Two new integration tests in
crates/nexum-engine/src/supervisor/tests.rssupervisor.rsstating the invariantexplicitly. Zero production-code changes — the property is already
structural (per-chain subscription filter on the dispatch hot path
regression guard so a future refactor cannot quietly break it.
Stacks on #42 (COW-1072 graceful shutdown). Linear: COW-1073.
The invariant
For any two modules A (chain X) and B (chain Y) with X ≠ Y:
path — the
[[subscription]] chain_idfilter indispatch_block/
dispatch_logexcludes B before any restart / poison logicruns.
LoadedModuleonly mutates A's own per-module state(
alive,failure_count,next_attempt,failure_timestamps,poisoned). B is untouched.reconnect task spawned by
open_block_streams/open_log_streams(COW-1071). Chain Y's task is independentand keeps yielding.
New tests
multi_chain_dispatch_isolates_modules_by_chainchain_id = 1andchain_id = 100.dispatched == 1, module-bunchanged.
dispatched == 1, module-aunchanged.
multi_chain_poisoned_module_does_not_affect_other_chainsPoisonPolicy::new(2, Duration::from_secs(60)).alive == false,poisoned == 0.dispatched == 1).poisoned == 1.(3), which needs E2E coverage — see Out of Scope below).
Wall-clock ≈ 1.2 s for the second test (single backoff window).
supervisor.rs docstring
The module-level comment now articulates the multi-chain isolation
invariant explicitly, with a forward-reference to the test names
that guard it. Future contributors reading the dispatch path will
see the property is intentional and load-bearing.
Out of scope
chain B keeps streaming would require mocking
ProviderPool::subscribe_blocks, which today goes through realalloy + tokio infrastructure. COW-1064 (E2E 4–6 h testnet
integration) and COW-1031 (7-day soak on Sepolia + Arbitrum
Sepolia) exercise this path against live RPCs.
reconnect policy is workspace-wide. Per-chain tuning is a 0.3
follow-up, not blocking M4.
Verification
cargo test --workspace→ 163 host tests + 6 doctests passing(+2 from the new tests; was 161 + 6).
cargo clippy --all-targets --workspace -- -D warningsclean.cargo fmt --all --checkclean.AI assistance disclosure
Claude (Opus 4.7, 1M-context) drafted the two integration tests
and the supervisor.rs docstring. I reviewed each line, ran the
full sanity sweep above, and confirmed both tests pass + fail
correctly (verified the second test by temporarily flipping
example's subscriptionchain_idto 1 — both modules thencaught the dispatch, the assertion that example stays untouched
during the first bomb trap fired, and I reverted).