docs(ops): production deployment guide (COW-1030)#45
Closed
brunota20 wants to merge 1 commit into
Closed
Conversation
Adds `docs/production.md` — the operator handbook the
production-hardening milestone has been pointing at since M2.
Sister doc to `docs/06-production-hardening.md`: the existing
file is the architecture / design rationale (resource model,
restart policy, RPC resilience, logging + metrics design); this
new one is the concrete operator handbook (unit files, backup
recipes, alert rules, runbook procedures). Cross-referenced
both ways.
## Sections
1. **Pre-flight checklist** — every box you need ticked before
the first start: release-mode binary, persistent state dir,
metrics on loopback, paid RPC, Prometheus + log pipeline,
on-call runbook reference.
2. **systemd unit** — full `/etc/systemd/system/shepherd.service`
with: dedicated `shepherd` user, SIGINT for graceful
shutdown (30 s timeout — covers the COW-1072 last-block
persistence path), `NoNewPrivileges`/`ProtectSystem=strict`,
2 G memory cap (defence in depth on top of wasmtime's 64
MiB / module), restart-on-failure with 5 s backoff. Install
recipe + journalctl tail snippet.
3. **Docker Compose** — interim Dockerfile (multi-stage Rust
build + Debian slim runtime, non-root, EXPOSE 9100,
tini PID 1) + compose stack with bundled Prometheus, host
loopback port mapping only, `stop_signal: SIGINT`,
`stop_grace_period: 30s`, /metrics-based healthcheck.
Marked interim because the official Dockerfile is a
separate tracking issue.
4. **redb backup** — three operationally-supported paths:
cold backup (systemctl stop + cp + start; byte-identical
on graceful shutdown), hot backup (SIGSTOP + cp + SIGCONT
pattern, safe because the on-disk format is consistent at
any commit boundary), restore + `Database::check_integrity()`.
Honest about the redb 2.6 surface — no in-process snapshot
API today; flagged the roadmap. Retention policy:
7 daily / 4 weekly / 12 monthly = ~2.3 GiB on a 100 MiB
store.
5. **Logs** — JSON shape on stdout, two-tier retention model
(7 d hot full debug, 90 d cold INFO-only on S3 Glacier),
Vector config sample for journald -> Loki/hot +
journald -> S3/cold split. Sizing estimate based on the
E2E run shape (5 modules × 1 dispatch/12 s ≈ 200 MiB/wk
INFO+DEBUG combined).
6. **RPC selection** — provider plan recommendations
(Alchemy/Infura/QuickNode tiers), capacity sizing per
chain (1 block sub + N log subs + M `eth_call`/block where
M grows with TWAP active orders), why public nodes are
non-starters.
7. **Metrics + scraping** — complete metric surface table
from `grep metrics::counter/histogram/gauge`:
`shepherd_event_latency_seconds`,
`shepherd_module_errors_total`,
`shepherd_module_restarts_total`,
`shepherd_module_poisoned`,
`shepherd_chain_request_total`,
`shepherd_cow_api_submit_total`,
`shepherd_stream_reconnects_total`. Label set verified
against the source. 15 s scrape interval recommended.
8. **Workload-class tuning** — light indexer / TWAP-style
polling / multi-chain swarm classes with concrete
fuel + memory numbers. Honest that the limits are
compile-time constants today and per-module overrides via
`[engine.limits]` are a 0.3 follow-up — operator can
change `runtime/limits.rs` constants and rebuild, or
ensure load fits within defaults.
9. **Alert rules** — full `prometheus-rules.yml` covering
the seven alerts that map to the metric surface:
- `ShepherdModulePoisoned` (page) — production module
quarantined, needs operator action.
- `ShepherdModuleTraps` (ticket) — pre-poison signal.
- `ShepherdRpcErrorRate` (ticket) — > 5% RPC errors.
- `ShepherdReconnectStorm` (ticket) — WS flapping.
- `ShepherdCowApiErrorRate` (ticket) — > 20% orderbook
errors over 15 min.
- `ShepherdDispatchLatency` (ticket) — p95 > 5 s
sustained.
- `ShepherdDown` (page) — engine absent for 2 min.
10. **Operational runbook** — five common tasks:
tail-per-module, reset poisoned module, add module to
running deploy, inspect local-store, bump log level.
Each task carries a concrete shell snippet.
11. **Pre-upgrade checklist** — CHANGELOG read, cold backup,
stage binary, validate `supervisor ready modules=N
chains=M`, swap binary, restart, watch 5 min.
12. **References** — links to the architecture doc, ADRs,
runbooks, sister Linear issues.
## Drive-by fix
The COW-1064 e2e report template (committed in PR #44) had two
metric-label mistakes I caught while writing the metric
surface table in section 7:
- `result="ok|err"` -> `outcome="ok|err"` (the actual label
name in `cow_api.rs` + `chain.rs`).
- `reason="trap"` -> `error_kind="trap"` (the actual label
name in `supervisor.rs`).
Both labels appear in 4 places in the template (section 5
metric delta table + section 7 acceptance checklist). Fixed
in-place rather than as a separate PR.
## Workspace impact
- No code changes.
- No new build dependencies.
- `cargo fmt --all --check` clean.
Linear: COW-1030. Eleventh M4 issue landed; stacks on #44
(COW-1064).
Author
|
Work landed via |
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…-1064 dry run
Reconfigures the M3 example modules' manifests to the pinned
identities for the 2026-06-18 COW-1064 E2E dry run (Bruno's test
EOA + Safe on Sepolia) and adds a `docs/operations/e2e-cow-1064-
prep.md` companion to the runbook that captures every
copy-paste-able value the operator needs to drive the on-chain
side of the run without re-deriving any UID, address, or
calldata.
## Module config pinning
`modules/examples/stop-loss/module.toml`:
- owner -> 0x7bF140727D27ea64b607E042f1225680B40ECa6A (test EOA)
- sell_token -> WETH9 Sepolia (was a mainnet KNC address — bug
that would have failed the orderbook accept regardless)
- buy_token -> COW Sepolia (verified on-chain: name="CoW
Protocol Token", symbol="COW", decimals=18)
- sell_amount -> 0.005 WETH (fits 0.01 WETH wrap budget)
- buy_amount -> 20 COW (conservative quote)
- trigger_price -> $2000 (above the Sepolia Chainlink mocked
answer ~$1681 so the strategy fires on the first block)
`modules/examples/balance-tracker/module.toml`:
- addresses -> EOA + Safe (was the hardhat default accounts)
- change_threshold -> 0.001 ETH (was 0.1; lower so the small
E2E gas-side transfers show as Warn diffs)
## OrderUid pinning + regression test
`modules/examples/stop-loss/src/strategy.rs` gains
`cow_1064_e2e_settings_yield_expected_uid`: an integration
test that constructs `Settings` from the exact same constants
as the new manifest and asserts the resulting `build_creation`
UID against:
0xc2b9cb4ea1ee5a86d8049ac09d8f494bf04cca0a68407285f31e2e6379800be8
7bf140727d27ea64b607e042f1225680b40eca6a
ffffffff
(orderDigest || owner || validTo per packOrderUidParams.)
If anything drifts — manifest values, EIP-712 type-hash,
domain separator — the test fires before the run starts, not
during the run.
## Run-prep punch list
`docs/operations/e2e-cow-1064-prep.md` (~ 280 lines):
1. **Pinned identities table** — every address the runbook
references (EOA, Safe, ComposableCoW, TWAP handler,
EthFlow, GPv2Settlement, GPv2VaultRelayer, WETH, COW
token, domain separator). All verified via `eth_getCode`
on Sepolia before commit.
2. **Per-module config pinning** — stop-loss + balance-
tracker effective values in table form.
3. **OrderUid decomposition** — orderDigest (32) + owner (20)
+ validTo (4) breakdown so an operator reading
`setPreSignature` calldata can sanity-check the UID
without redoing the EIP-712 math.
4. **Four on-chain actions** — each as a numbered step with
the exact contract + function + arguments + Etherscan
write-UI URL:
- Action 1: wrap 0.01 ETH -> 0.01 WETH9 (optional, only
for `submitted:` path; `backoff:` works without).
- Action 2: setPreSignature + WETH allowance to
GPv2VaultRelayer (optional, paired with action 1).
- Action 3: TWAP create() via Safe TX Builder; the full
516-byte calldata pinned verbatim (selector 0x6bfae1ca
+ tuple-encoded ConditionalOrderParams + dispatch=true).
- Action 4: EthFlow swap via cow-swap UI on Sepolia
(UI-driven for the quote endpoint hit; calldata
fallback link if UI flakes).
5. **Validation snippets** — `cast` invocations to check EOA
+ Safe balances, WETH balance, allowance,
`preSignature(bytes)` lookup, and a `journalctl + jq`
one-liner that tails per-module terminal markers in real
time.
6. **Re-derivation recipes** — Python + `cargo test`
commands to regenerate every pinned value if config drift
ever forces a re-run with different identities.
7. **Per-run acceptance checklist** — 9 box-checks that
double-pin section 7 of the e2e-report template, scoped
to THIS specific run.
## Workspace impact
- `cargo test -p stop-loss --lib` -> 8 passed (was 7; +1
for the new pinning test).
- `cargo fmt --all --check` clean.
- No production-code changes outside the test module.
Linear: COW-1064. Twelfth M4 deliverable; stacks on #45.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…-1064 dry run
Reconfigures the M3 example modules' manifests to the pinned
identities for the 2026-06-18 COW-1064 E2E dry run (Bruno's test
EOA + Safe on Sepolia) and adds a `docs/operations/e2e-cow-1064-
prep.md` companion to the runbook that captures every
copy-paste-able value the operator needs to drive the on-chain
side of the run without re-deriving any UID, address, or
calldata.
## Module config pinning
`modules/examples/stop-loss/module.toml`:
- owner -> 0x7bF140727D27ea64b607E042f1225680B40ECa6A (test EOA)
- sell_token -> WETH9 Sepolia (was a mainnet KNC address — bug
that would have failed the orderbook accept regardless)
- buy_token -> COW Sepolia (verified on-chain: name="CoW
Protocol Token", symbol="COW", decimals=18)
- sell_amount -> 0.005 WETH (fits 0.01 WETH wrap budget)
- buy_amount -> 20 COW (conservative quote)
- trigger_price -> $2000 (above the Sepolia Chainlink mocked
answer ~$1681 so the strategy fires on the first block)
`modules/examples/balance-tracker/module.toml`:
- addresses -> EOA + Safe (was the hardhat default accounts)
- change_threshold -> 0.001 ETH (was 0.1; lower so the small
E2E gas-side transfers show as Warn diffs)
## OrderUid pinning + regression test
`modules/examples/stop-loss/src/strategy.rs` gains
`cow_1064_e2e_settings_yield_expected_uid`: an integration
test that constructs `Settings` from the exact same constants
as the new manifest and asserts the resulting `build_creation`
UID against:
0xc2b9cb4ea1ee5a86d8049ac09d8f494bf04cca0a68407285f31e2e6379800be8
7bf140727d27ea64b607e042f1225680b40eca6a
ffffffff
(orderDigest || owner || validTo per packOrderUidParams.)
If anything drifts — manifest values, EIP-712 type-hash,
domain separator — the test fires before the run starts, not
during the run.
## Run-prep punch list
`docs/operations/e2e-cow-1064-prep.md` (~ 280 lines):
1. **Pinned identities table** — every address the runbook
references (EOA, Safe, ComposableCoW, TWAP handler,
EthFlow, GPv2Settlement, GPv2VaultRelayer, WETH, COW
token, domain separator). All verified via `eth_getCode`
on Sepolia before commit.
2. **Per-module config pinning** — stop-loss + balance-
tracker effective values in table form.
3. **OrderUid decomposition** — orderDigest (32) + owner (20)
+ validTo (4) breakdown so an operator reading
`setPreSignature` calldata can sanity-check the UID
without redoing the EIP-712 math.
4. **Four on-chain actions** — each as a numbered step with
the exact contract + function + arguments + Etherscan
write-UI URL:
- Action 1: wrap 0.01 ETH -> 0.01 WETH9 (optional, only
for `submitted:` path; `backoff:` works without).
- Action 2: setPreSignature + WETH allowance to
GPv2VaultRelayer (optional, paired with action 1).
- Action 3: TWAP create() via Safe TX Builder; the full
516-byte calldata pinned verbatim (selector 0x6bfae1ca
+ tuple-encoded ConditionalOrderParams + dispatch=true).
- Action 4: EthFlow swap via cow-swap UI on Sepolia
(UI-driven for the quote endpoint hit; calldata
fallback link if UI flakes).
5. **Validation snippets** — `cast` invocations to check EOA
+ Safe balances, WETH balance, allowance,
`preSignature(bytes)` lookup, and a `journalctl + jq`
one-liner that tails per-module terminal markers in real
time.
6. **Re-derivation recipes** — Python + `cargo test`
commands to regenerate every pinned value if config drift
ever forces a re-run with different identities.
7. **Per-run acceptance checklist** — 9 box-checks that
double-pin section 7 of the e2e-report template, scoped
to THIS specific run.
## Workspace impact
- `cargo test -p stop-loss --lib` -> 8 passed (was 7; +1
for the new pinning test).
- `cargo fmt --all --check` clean.
- No production-code changes outside the test module.
Linear: COW-1064. Twelfth M4 deliverable; stacks on #45.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…-1064 dry run
Reconfigures the M3 example modules' manifests to the pinned
identities for the 2026-06-18 COW-1064 E2E dry run (Bruno's test
EOA + Safe on Sepolia) and adds a `docs/operations/e2e-cow-1064-
prep.md` companion to the runbook that captures every
copy-paste-able value the operator needs to drive the on-chain
side of the run without re-deriving any UID, address, or
calldata.
## Module config pinning
`modules/examples/stop-loss/module.toml`:
- owner -> 0x7bF140727D27ea64b607E042f1225680B40ECa6A (test EOA)
- sell_token -> WETH9 Sepolia (was a mainnet KNC address — bug
that would have failed the orderbook accept regardless)
- buy_token -> COW Sepolia (verified on-chain: name="CoW
Protocol Token", symbol="COW", decimals=18)
- sell_amount -> 0.005 WETH (fits 0.01 WETH wrap budget)
- buy_amount -> 20 COW (conservative quote)
- trigger_price -> $2000 (above the Sepolia Chainlink mocked
answer ~$1681 so the strategy fires on the first block)
`modules/examples/balance-tracker/module.toml`:
- addresses -> EOA + Safe (was the hardhat default accounts)
- change_threshold -> 0.001 ETH (was 0.1; lower so the small
E2E gas-side transfers show as Warn diffs)
## OrderUid pinning + regression test
`modules/examples/stop-loss/src/strategy.rs` gains
`cow_1064_e2e_settings_yield_expected_uid`: an integration
test that constructs `Settings` from the exact same constants
as the new manifest and asserts the resulting `build_creation`
UID against:
0xc2b9cb4ea1ee5a86d8049ac09d8f494bf04cca0a68407285f31e2e6379800be8
7bf140727d27ea64b607e042f1225680b40eca6a
ffffffff
(orderDigest || owner || validTo per packOrderUidParams.)
If anything drifts — manifest values, EIP-712 type-hash,
domain separator — the test fires before the run starts, not
during the run.
## Run-prep punch list
`docs/operations/e2e-cow-1064-prep.md` (~ 280 lines):
1. **Pinned identities table** — every address the runbook
references (EOA, Safe, ComposableCoW, TWAP handler,
EthFlow, GPv2Settlement, GPv2VaultRelayer, WETH, COW
token, domain separator). All verified via `eth_getCode`
on Sepolia before commit.
2. **Per-module config pinning** — stop-loss + balance-
tracker effective values in table form.
3. **OrderUid decomposition** — orderDigest (32) + owner (20)
+ validTo (4) breakdown so an operator reading
`setPreSignature` calldata can sanity-check the UID
without redoing the EIP-712 math.
4. **Four on-chain actions** — each as a numbered step with
the exact contract + function + arguments + Etherscan
write-UI URL:
- Action 1: wrap 0.01 ETH -> 0.01 WETH9 (optional, only
for `submitted:` path; `backoff:` works without).
- Action 2: setPreSignature + WETH allowance to
GPv2VaultRelayer (optional, paired with action 1).
- Action 3: TWAP create() via Safe TX Builder; the full
516-byte calldata pinned verbatim (selector 0x6bfae1ca
+ tuple-encoded ConditionalOrderParams + dispatch=true).
- Action 4: EthFlow swap via cow-swap UI on Sepolia
(UI-driven for the quote endpoint hit; calldata
fallback link if UI flakes).
5. **Validation snippets** — `cast` invocations to check EOA
+ Safe balances, WETH balance, allowance,
`preSignature(bytes)` lookup, and a `journalctl + jq`
one-liner that tails per-module terminal markers in real
time.
6. **Re-derivation recipes** — Python + `cargo test`
commands to regenerate every pinned value if config drift
ever forces a re-run with different identities.
7. **Per-run acceptance checklist** — 9 box-checks that
double-pin section 7 of the e2e-report template, scoped
to THIS specific run.
## Workspace impact
- `cargo test -p stop-loss --lib` -> 8 passed (was 7; +1
for the new pinning test).
- `cargo fmt --all --check` clean.
- No production-code changes outside the test module.
Linear: COW-1064. Twelfth M4 deliverable; stacks on #45.
brunota20
added a commit
that referenced
this pull request
Jun 25, 2026
…-1064 dry run
Reconfigures the M3 example modules' manifests to the pinned
identities for the 2026-06-18 COW-1064 E2E dry run (Bruno's test
EOA + Safe on Sepolia) and adds a `docs/operations/e2e-cow-1064-
prep.md` companion to the runbook that captures every
copy-paste-able value the operator needs to drive the on-chain
side of the run without re-deriving any UID, address, or
calldata.
## Module config pinning
`modules/examples/stop-loss/module.toml`:
- owner -> 0x7bF140727D27ea64b607E042f1225680B40ECa6A (test EOA)
- sell_token -> WETH9 Sepolia (was a mainnet KNC address — bug
that would have failed the orderbook accept regardless)
- buy_token -> COW Sepolia (verified on-chain: name="CoW
Protocol Token", symbol="COW", decimals=18)
- sell_amount -> 0.005 WETH (fits 0.01 WETH wrap budget)
- buy_amount -> 20 COW (conservative quote)
- trigger_price -> $2000 (above the Sepolia Chainlink mocked
answer ~$1681 so the strategy fires on the first block)
`modules/examples/balance-tracker/module.toml`:
- addresses -> EOA + Safe (was the hardhat default accounts)
- change_threshold -> 0.001 ETH (was 0.1; lower so the small
E2E gas-side transfers show as Warn diffs)
## OrderUid pinning + regression test
`modules/examples/stop-loss/src/strategy.rs` gains
`cow_1064_e2e_settings_yield_expected_uid`: an integration
test that constructs `Settings` from the exact same constants
as the new manifest and asserts the resulting `build_creation`
UID against:
0xc2b9cb4ea1ee5a86d8049ac09d8f494bf04cca0a68407285f31e2e6379800be8
7bf140727d27ea64b607e042f1225680b40eca6a
ffffffff
(orderDigest || owner || validTo per packOrderUidParams.)
If anything drifts — manifest values, EIP-712 type-hash,
domain separator — the test fires before the run starts, not
during the run.
## Run-prep punch list
`docs/operations/e2e-cow-1064-prep.md` (~ 280 lines):
1. **Pinned identities table** — every address the runbook
references (EOA, Safe, ComposableCoW, TWAP handler,
EthFlow, GPv2Settlement, GPv2VaultRelayer, WETH, COW
token, domain separator). All verified via `eth_getCode`
on Sepolia before commit.
2. **Per-module config pinning** — stop-loss + balance-
tracker effective values in table form.
3. **OrderUid decomposition** — orderDigest (32) + owner (20)
+ validTo (4) breakdown so an operator reading
`setPreSignature` calldata can sanity-check the UID
without redoing the EIP-712 math.
4. **Four on-chain actions** — each as a numbered step with
the exact contract + function + arguments + Etherscan
write-UI URL:
- Action 1: wrap 0.01 ETH -> 0.01 WETH9 (optional, only
for `submitted:` path; `backoff:` works without).
- Action 2: setPreSignature + WETH allowance to
GPv2VaultRelayer (optional, paired with action 1).
- Action 3: TWAP create() via Safe TX Builder; the full
516-byte calldata pinned verbatim (selector 0x6bfae1ca
+ tuple-encoded ConditionalOrderParams + dispatch=true).
- Action 4: EthFlow swap via cow-swap UI on Sepolia
(UI-driven for the quote endpoint hit; calldata
fallback link if UI flakes).
5. **Validation snippets** — `cast` invocations to check EOA
+ Safe balances, WETH balance, allowance,
`preSignature(bytes)` lookup, and a `journalctl + jq`
one-liner that tails per-module terminal markers in real
time.
6. **Re-derivation recipes** — Python + `cargo test`
commands to regenerate every pinned value if config drift
ever forces a re-run with different identities.
7. **Per-run acceptance checklist** — 9 box-checks that
double-pin section 7 of the e2e-report template, scoped
to THIS specific run.
## Workspace impact
- `cargo test -p stop-loss --lib` -> 8 passed (was 7; +1
for the new pinning test).
- `cargo fmt --all --check` clean.
- No production-code changes outside the test module.
Linear: COW-1064. Twelfth M4 deliverable; stacks on #45.
lgahdl
added a commit
that referenced
this pull request
Jul 10, 2026
The old harness mocked submit_order and classified success by whether
the strategy POSTed an order. The redesigned ethflow-watcher strategy
GETs /api/v1/orders/{uid} instead — so every fixture previously
classified as RejectedUnexpected and the binary exited 1.
Now programs a 200 catch-all for cow_api_request, classifies success
by whether observed:{uid} was written to local store, and renames the
Submitted classification to Observed throughout.
Closes #45
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lgahdl
added a commit
that referenced
this pull request
Jul 10, 2026
Review follow-ups on top of the rebase onto develop:
- classify_ok now demands the full observe contract, not just the
store marker: zero submit_order calls and exactly one
GET /api/v1/orders/{uid} against the fixture's UID. A regression
that writes the marker after a different call shape (or that also
submits) is classified RejectedUnexpected instead of Observed, so
the backtest stays an independent check on the module tests.
- Regenerated docs/operations/backtest-reports/backtest-7d-2026-06-22.md
from the committed fixtures: 240/240 (100.0%) Observed, replacing the
stale submit-era report (issue #45's "stale and unreproducible" item).
Verified: cargo run -p shepherd-backtest -- --fixtures
tools/backtest-collect/fixtures-2026-06-22.json exits 0 with 240/240.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfSsJPYDQ1GQGbnSVFxATx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
docs/production.md— the operator handbook theproduction-hardening milestone has been pointing at since M2.
Sister doc to
docs/06-production-hardening.md: that existingfile is the architecture / design rationale; this is the
concrete operator handbook (unit files, backup recipes, alert
rules, runbook procedures). Cross-referenced both ways so an
operator reading either lands on the right next step.
Stacks on #44 (COW-1064 E2E scaffold). Linear: COW-1030.
Sections (12 total)
dir, metrics on loopback, paid RPC, observability pipeline.
shepherd.servicewith dedicateduser, SIGINT for graceful shutdown (30 s timeout covers
the COW-1072 last-block-persistence path), filesystem
hardening, 2 G memory cap (defence in depth over wasmtime's
per-module 64 MiB).
with bundled Prometheus, loopback-only port mapping,
stop_signal: SIGINT,/metrics-based healthcheck. Markedinterim because the official Dockerfile is tracked
separately.
(cold/hot/restore + integrity check) with the redb 2.6
limitations stated honestly. Retention policy: 7 daily / 4
weekly / 12 monthly ≈ 2.3 GiB on a 100 MiB store.
full debug + 90 d cold INFO-only on S3 Glacier), Vector
config sample for the split. Sizing based on the E2E run
shape.
capacity sizing, why public nodes are non-starters.
verified against
grep metrics::(counter|histogram|gauge)!:shepherd_event_latency_seconds{module, event_kind}shepherd_module_errors_total{module, error_kind}shepherd_module_restarts_total{module}shepherd_module_poisoned{module}shepherd_chain_request_total{chain_id, method, outcome}shepherd_cow_api_submit_total{chain_id, outcome}shepherd_stream_reconnects_total{kind, chain_id, module?}multi-chain swarm classes with concrete fuel + memory
numbers. Honest that limits are compile-time constants
today and per-module overrides via
[engine.limits]are a0.3 follow-up.
prometheus-rules.yml(7 alerts,page vs ticket severity).
ShepherdModulePoisoned+ShepherdDownpage; the rest ticket.reset poisoned module, add module to running deploy,
inspect local-store, bump log level. Each carries a
concrete shell snippet.
promoting.
runbooks, sister Linear issues.
Drive-by fix
The COW-1064 e2e report template (committed in #44) had two
metric-label mistakes I caught while writing this PR's section
7 metric surface table:
result=\"ok|err\"->outcome=\"ok|err\"(the actual labelname in
cow_api.rs+chain.rs).reason=\"trap\"->error_kind=\"trap\"(the actual labelname in
supervisor.rs).Both labels appear in 4 places in the template; fixed
in-place rather than as a separate PR.
Workspace impact
cargo fmt --all --checkclean.AI assistance disclosure
Claude (Opus 4.7, 1M-context) drafted
docs/production.mdworking from: the issue's spec (systemd / Docker Compose /
redb backup / log retention / workload tuning / alert
templates), the existing
docs/06-production-hardening.mdshape, the actual metric surface (verified via
grep metrics::(counter|histogram|gauge)!over the engine crate),the actual runtime constants (
DEFAULT_FUEL_PER_EVENT,DEFAULT_MEMORY_LIMIT,POISON_MAX_FAILURES,POISON_WINDOW,RESTART_MAX_BACKOFF), and the existingtestnet runbooks for shape consistency. I reviewed every
section, verified each shell snippet would work as written
(systemd directives, docker compose schema, journalctl + jq
filter syntax, redb integrity API), and the drive-by metric-
label fix in the e2e report template came from cross-checking
the table against the source.