feat(ops): E2E testnet integration scaffold (COW-1064)#44
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This is the integration step between unit-test
coverage (MockHost, per-module strategy tests) and the COW-1031
7-day soak; the soak validates stability, this validates
correctness in a live dispatch context.
## What this PR ships (scaffold, not run)
- `engine.e2e.toml` — unified Sepolia config loading all 5
modules (twap-monitor + ethflow-watcher + price-alert +
balance-tracker + stop-loss), separate `state_dir = ./data/e2e`,
Prometheus `/metrics` enabled on 127.0.0.1:9100. Operator
swaps in their Alchemy/Infura WS URL before launching the run.
- `justfile` targets `build-e2e` + `run-e2e`. `build-e2e`
reuses `build-m2 + build-m3` so the 5 wasm artefacts are
produced in one go; `run-e2e` boots the engine pointed at
`engine.e2e.toml` (no `--pretty-logs` so production-shape
JSON logs are emitted, ready for jq mining).
- `docs/operations/e2e-testnet-runbook.md` — full operator
runbook mirroring the M2 + M3 shape. Sections cover RPC
selection, on-chain prep (test EOA + Safe + stop-loss
pre-sign), boot sequence + expected log shape, the three
on-chain triggers that satisfy the per-module terminal-state
markers, metrics capture, red flags to watch, and report
filing. Acceptance bar from COW-1064 reproduced verbatim.
- `docs/operations/e2e-reports/e2e-report.template.md` — empty
report skeleton operator copies to
`e2e-report-YYYY-MM-DD.md` at the start of each run and
fills in as the run progresses. Sections: run metadata,
chain coverage, on-chain actions submitted, per-module
terminal-state markers, error-count deltas from Prometheus,
anomalies, acceptance checklist, sign-off.
## What this PR explicitly does NOT do
The 4-6 h run itself is operator-driven and cannot be
exercised from CI:
1. Real Sepolia RPC keys (rate-limited public node will not
survive a multi-hour run with 4+ eth_call per block).
2. Funded test EOA + ComposableCoW Safe access to submit a
real conditional order (twap-monitor's only path to a
`submitted:` marker).
3. EthFlow swap from a real EOA on Sepolia
(ethflow-watcher's only path to a `submitted:` marker).
4. `setPreSignature` + sell-token allowance from the stop-loss
`owner` EOA (stop-loss's only path to a `submitted:`
marker that is not a typed `TransferSimulationFailed` warn).
5. 4-6 h wall clock + metrics-start.txt / metrics-end.txt
capture.
The runbook is unambiguous about what each step requires; the
report template's section 8 is the gating sign-off for
COW-1031 (7-day soak).
## Smoke-validation done before commit
Booted `engine.e2e.toml` end-to-end against live Sepolia for
60+ s (kill -INT-style early shutdown):
```
INFO supervisor ready modules=5 chains=1
INFO log subscription open module=twap-monitor chain_id=11155111
INFO block subscription open chain_id=11155111
INFO log subscription open module=ethflow-watcher chain_id=11155111
DEBUG dispatch ok module=twap-monitor block_number=11088259 latency_ms=1
WARN price-alert: TRIGGERED answer=168110190000 threshold=250000000000 (Below)
DEBUG dispatch ok module=balance-tracker block_number=11088259 latency_ms=271
WARN stop-loss retry on next block (0): orderbook error
(TransferSimulationFailed): sell token cannot be transferred
DEBUG dispatch ok module=stop-loss block_number=11088259 latency_ms=1802
```
This proves: 5/5 modules init successfully, both log
subscriptions + the block subscription open, the dispatch loop
ticks against real Sepolia blocks, every module that has a
block subscription dispatches on every block, and the real
RPC + Chainlink decode + cow-api submit path is exercised
inside seconds. The remaining acceptance bar (terminal
markers on twap-monitor + ethflow-watcher, 1500-block run,
0-ERROR supervisor log) only the operator can produce.
## Workspace impact
- No production-code changes (this is pure ops scaffolding).
- `cargo fmt --all --check` clean.
- `cargo build --target wasm32-wasip2 --release` produces all
5 module artefacts named exactly as `engine.e2e.toml`
references them (twap_monitor.wasm, ethflow_watcher.wasm,
price_alert.wasm, balance_tracker.wasm, stop_loss.wasm).
Linear: COW-1064. Tenth M4 issue landed; stacks on #43 (COW-1073).
This was referenced Jun 18, 2026
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the engine config, runbook, and report template for the
4-6 h E2E run on Sepolia with all 5 modules dispatched
simultaneously. This PR is scaffold-only — the run itself is
operator-driven (real Sepolia RPC keys, funded test EOA,
on-chain order submissions, 4-6 h wall clock) and cannot be
exercised from CI.
Stacks on #43 (COW-1073 multi-chain isolation). Linear: COW-1064.
What ships
engine.e2e.toml[engine.metrics] enabled = trueon 127.0.0.1:9100 + separatestate_dir = ./data/e2e.justfilebuild-e2e(reusesbuild-m2 + build-m3) andrun-e2etargets. JSON logs (no--pretty-logs) for jq mining.docs/operations/e2e-testnet-runbook.mddocs/operations/e2e-reports/e2e-report.template.md/metrics, anomalies, acceptance checklist, sign-off (the gating decision for COW-1031).What this PR does NOT do
The 4-6 h run itself is operator-driven and requires:
throttles
eth_subscribeandeth_callunder sustainedload (≥ 4 calls/block from price-alert + balance-tracker ×2 +
stop-loss + 1/registered-order from twap-monitor).
twap-monitor's only path to a
submitted:marker.ethflow-watcher's only path to a
submitted:marker.setPreSignature+ sell-token allowance from the stop-lossownerEOA — stop-loss's only path to asubmitted:marker that is not a typed
TransferSimulationFailedwarn.metrics-start.txt/metrics-end.txtcapture + the report's section 8 sign-off, which gates
COW-1031 (7-day soak).
The runbook is explicit about each step; the report template's
acceptance checklist makes the closure conditions for COW-1064
unambiguous.
Smoke-validation done before commit
Booted
engine.e2e.tomlend-to-end against live Sepolia for60+ s. Observed:
5/5 modules init successfully, both log subscriptions + the
block subscription open, dispatch loop ticks against real
Sepolia blocks, every block-subscribing module dispatches every
block, and the real RPC + Chainlink decode + cow-api submit
path is exercised inside seconds. Acceptance markers for
ethflow-watcher + the on-chain
submitted:paths require theoperator's on-chain actions.
Workspace impact
cargo fmt --all --checkclean.engine.e2e.tomlreferences (twap_monitor.wasm,ethflow_watcher.wasm,price_alert.wasm,balance_tracker.wasm,stop_loss.wasm).AI assistance disclosure
Claude (Opus 4.7, 1M-context) drafted the runbook, report
template, and
engine.e2e.tomlfrom the existing M2 + M3runbook shapes. I reviewed each file, ran the 60 s smoke boot
against live Sepolia (output above), and confirmed the
acceptance bar's "scaffold half" is complete. The "run half" I
will execute manually before pushing the COW-1064 report PR.