feat(soak): add 7-day soak harness (engine.soak.toml, runner, snapshots, runbook)#345
Merged
Conversation
mfw78
added
the
base: develop
…ts, runbook) Adds the tooling gap that was blocking #65/#66 grant evidence: - engine.soak.toml: soak-tuned config (info log level, separate state_dir) - scripts/soak-run.sh: launches engine + hourly metrics snapshot loop - scripts/soak-finish.sh: graceful shutdown, stops snapshot loop, final metrics - docs/operations/soak-runbook.md: operator guide with evidence artifact map M1 (24h), M2 (48h), and M4 (7-day) evidence artifacts fall out of the hourly snapshot loop automatically. Closes #122
- Add '# shellcheck shell=bash' + file-level SC2034 disable to lib.sh (constants are used by callers that source the file; shellcheck cannot follow the source chain without explicit directives) - Replace ls glob with find in soak-finish.sh snapshot count (SC2012)
- disown snapshot loop subshell after launch so it survives SSH disconnect — a naked () & dies when the terminal session ends, which would silently drop all hourly evidence snapshots - supervisor-ready regex: replace [^}]* with .* to avoid stopping at nested JSON object braces (e.g. tracing span fields); add [^0-9] anchors to avoid matching "modules":50 as "modules":5 - rpc_url substitution regex: allow inline comments on the section header ([chains.11155111] # Sepolia) by using [^\n]* instead of a bare \n after the bracket - add scripts/.state.soak to .gitignore alongside scripts/.state
…idecar
The 160-line soak-run.sh + soak-finish.sh are replaced by two Docker
services in docker-compose.soak.yml:
engine — nexum binary with restart: unless-stopped and Docker's
json-file log rotation (500 MB × 14 ≈ 7 GB cap), removing
the nohup/disown/PID-file/regex-supervisor-ready dance.
snapshotter — Alpine container running scripts/soak-snapshot.sh (10
lines): captures metrics-start on boot, then hourly
metrics-snap-*.txt to the bind-mounted evidence directory.
engine.soak.docker.toml is a new Docker-compatible variant of
engine.soak.toml: Docker paths (/opt/shepherd/modules/), SEPOLIA_RPC_URL
env-var substitution (no render_soak_config Python step needed), and
bind_addr = "0.0.0.0:9100" so the snapshotter reaches it via the Docker
bridge network.
- Delete scripts/soak-run.sh, scripts/soak-finish.sh
- Remove soak state helpers from scripts/lib.sh (render_soak_config,
SOAK_STATE_FILE and the write/clear/value functions)
- Rewrite docs/operations/soak-runbook.md for the docker-compose workflow
- engine.soak.toml kept for direct-binary use; updated comment
…evidence
Review follow-ups before the July 14-17 soak window:
- soak-snapshot.sh: BusyBox wget instead of an apk install at every
container (re)start; scrape via temp file + mv so a failed scrape
never leaves an empty evidence file; retry the baseline scrape so a
transient hiccup cannot exit the script and write a spurious
metrics-start file on the restart-policy rerun.
- runbook pre-flight: ghcr.io/nullislabs/shepherd:latest is only
published from main and does not exist yet - the image step now
mandates either a local build pinned to the soak commit or a CI
workflow_dispatch publish, and records the image digest in the
evidence set.
- runbook monitoring: the per-module jq filter was doubly broken
(compose log prefixes are not JSON; fields are flattened to top
level, not under .fields). Replaced with a working docker logs +
.message filter.
- evidence: hourly host-side docker stats loop (memory.log - the
exporter has no process collector, so nothing else proves memory
stayed flat) and a docker inspect restart record (RestartCount /
OOMKilled) so a 3am crash-restart is visible in the artifact set
instead of surfacing as unexplained counter resets.
- compose: ${SEPOLIA_RPC_URL:?} guard so `up` refuses fast instead of
booting an engine that crash-loops on the unresolvable config var.
- engine.soak.toml: header referenced a render_soak_config helper that
does not exist; replaced with the real manual-run commands (the
engine substitutes ${VAR} natively). Dropped the dead
SOAK_REPORTS_DIR from lib.sh.
- runbook restart section: both services auto-restart; corrected the
stale shell-runner-era semantics and the engine.log guidance
(compress, do not commit ~7 GB into the repo).
Smoke test finding: the Prometheus exporter serves a 200 with an empty body until the first counter increments after boot, so the baseline scrape landed as a 0-byte file. Require a non-empty body before accepting any snapshot - the retry loop then holds until the first real metrics render. Smoke-tested: local image build + compose up; engine healthy, modules dispatching (14 eth_calls at t+80s), metrics-start-*.txt written with 4 KB of real counter values.
mfw78
force-pushed
the
feat/122-soak-harness-clean
branch
from
038cc56 to
5cdaf33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the tooling gap blocking #65/#66 grant evidence (hard gate: soak running clean by Jul 21). The harness pivoted from shell-runner scripts to Docker Compose during review — Docker owns crash recovery and log rotation instead of hand-rolled supervision:
docker-compose.soak.yml— two services: engine (restart: unless-stopped, SIGINT graceful stop, json-file log rotation capped at ~7 GB, 2 GB / 2 CPU limits, healthcheck on the metrics port,${SEPOLIA_RPC_URL:?}fail-fast guard) and snapshotter (Alpine sidecar).scripts/soak-snapshot.sh— 10-line BusyBox sidecar: baseline scrape with retry, then hourly/metricssnapshots written atomically (temp file +mv, so a failed scrape never leaves an empty evidence file).engine.soak.docker.toml/engine.soak.toml— Docker and manual-run configs:infolog level,data/soakstate dir, same 5 modules as e2e. The manual config documents the real build +nexum --engine-configcommands.docs/operations/soak-runbook.md— operator guide: pre-flight checklist (including the mandatory image pin —ghcr.io/nullislabs/shepherd:latestis only published frommainand does not exist yet; build locally orgh workflow run docker.yml --ref develop), start/stop/monitor commands, evidence artifact table (M1/M2/M4), troubleshooting.docker statsmemory log (no process collector in the exporter, so nothing else proves memory stayed flat),docker inspectrestart record (RestartCount/OOMKilled), image digest pin, compressed engine log.Closes evidence: add 7-day soak harness with periodic metric snapshots #122
Test plan
docker compose -f docker-compose.soak.yml config— valid; refuses with a clear error whenSEPOLIA_RPC_URLis unsetshellcheck scripts/soak-snapshot.sh— cleandocker build+up, healthcheck green,metrics-startfile written) — see review thread