Skip to content

feat(chain): cap JSON-RPC response size before lowering into the guest#309

Closed
lgahdl wants to merge 9 commits into
dev/m1from
feat/154-chain-response-size-cap
Closed

feat(chain): cap JSON-RPC response size before lowering into the guest#309
lgahdl wants to merge 9 commits into
dev/m1from
feat/154-chain-response-size-cap

Conversation

@lgahdl

@lgahdl lgahdl commented Jul 9, 2026 •
edited by mfw78
Loading

Copy link
Copy Markdown
Collaborator

What

Adds a host-side cap on chain JSON-RPC response bodies so a misbehaving or adversarial node cannot saturate the guest heap via a single large reply.

engine_config.rs gains a new [limits.chain] TOML section with a response_body_max_bytes knob (default 1 MiB), resolved through ModuleLimits::chain_response_max_bytes() with the same zero-saturates-to-one handling as the http/logs/poison sections.

HostState carries a chain_response_max_bytes: usize, threaded from config at boot in supervisor.rs and preserved across restarts via LoadedModule.

host/impls/chain.rs applies check_response_cap() to request() after receiving the provider response; a rejection logs a warning, increments the shepherd_chain_response_capped_total metric, and returns ChainError::Fault(InvalidInput). The request-batch path also tracks a running aggregate total against the same cap, so a wide batch of individually-legal bodies cannot lower an unbounded Vec<RpcResult> into guest memory in one go.

Subscription-delivered payloads (dispatch_chain_log) are not covered by this cap; that path lowers eth_subscribe notifications rather than chain::request responses, and bounding it is left as a follow-up rather than widened into this PR.

Why

A chain node (or anything on-path to it) that returns an oversized JSON-RPC response body can otherwise force the guest to buffer an arbitrarily large payload, giving an adversarial or misbehaving node a way to exhaust guest memory. Capping the response host-side, before it is copied into the guest, closes that gap the same way the existing [limits.http] cap does for outbound HTTP.

Testing

  • chain_limits_default_when_absent, chain_limits_parse_with_override, chain_limits_saturate_degenerate_zero in engine_config.rs cover config resolution.
  • response_at_cap_is_accepted and response_over_cap_returns_invalid_input in host::impls::chain::tests cover the per-entry cap boundary.
  • An end-to-end harness test in test_utils/harness.rs drives the cap through the real config -> HostState -> chain::request wiring with the price-alert module.
  • engine.example.toml documents the new [limits.chain] section.

AI Assistance

Claude (Opus) used for the implementation.

Closes #154

@lgahdl lgahdl requested a review from mfw78 as a code owner July 9, 2026 21:41
@lgahdl lgahdl changed the base branch from dev/m1 to develop July 9, 2026 21:59
@lgahdl lgahdl changed the base branch from develop to feat/m1-mock-venue July 9, 2026 22:01
@lgahdl lgahdl force-pushed the feat/154-chain-response-size-cap branch 2 times, most recently from 4acf3ed to fa79d01 Compare July 9, 2026 22:04
@mfw78 mfw78 added the pr-train Part of the current stacked PR train; lands in sequence, do not merge out of order. label Jul 9, 2026
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from df69180 to d01c759 Compare July 14, 2026 02:19
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from 265f467 to b8aeb1c Compare July 14, 2026 02:19
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from d01c759 to fa01f17 Compare July 14, 2026 03:33
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from b8aeb1c to d94a4bd Compare July 14, 2026 03:33
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from fa01f17 to eaa7777 Compare July 14, 2026 04:47
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from 6db87ec to b4734ce Compare July 14, 2026 07:45
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from eaa7777 to 0e74b87 Compare July 14, 2026 07:45
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from b4734ce to e15aa66 Compare July 14, 2026 09:33
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch 2 times, most recently from cdf84b1 to fa87a85 Compare July 14, 2026 10:05
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from 4c508ec to 6f83ed6 Compare July 14, 2026 12:25
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from fa87a85 to 52ca638 Compare July 14, 2026 12:26
mfw78 and others added 9 commits July 14, 2026 12:47
@mfw78
feat(sdk): add the keeper conditional-source seam, retry ledger, and run
4e1ec36 
Introduce the world-neutral half of the poll loop in
nexum_sdk::keeper: a ConditionalSource trait (one watch in, one
outcome out, generic over the host, no venue-transport
pre-abstraction), a Tick carrying the dispatch instant, and the
RetryAction (try-next-block / backoff / drop) whose effects the
Retrier runs through the gate and watch-set stores. RetryAction
moves out of shepherd-sdk; the CoW crate re-exports it so no module
rewires.

shepherd-sdk keeps classify_api_error as the CoW errorType table but
returns the keeper action, and encodes two fixes in the one
classification path: classify_poll_error maps an unrecognised revert
selector with a structured payload to DontTryAgain instead of
re-polling every block forever, and DuplicatedOrder (both spellings)
classifies as already-submitted - the run records the
submitted: receipt and keeps the watch rather than dropping every
future tranche. classify_submit_error widens the table to the whole
CowApiError surface and gives Backoff its producer: a rate-limit
fault with server guidance gates the watch on the epoch clock.

cow::run composes the loop: watch-set scan -> gate check ->
source poll -> PollOutcome dispatch, driving submission through the
existing CowApiHost seam with the journal as the idempotency guard
and the retry ledger as the failure dispatch. Acceptance tests run
against the composed shepherd-sdk-test MockHost as integration
tests; no module is rewired yet.
@mfw78
docs(sdk): disambiguate the run intra-doc link
06c9b07
@mfw78
fix(cow): build the classify_poll_error test RpcError data as Bytes
149c7d2 
RpcError.data is now Bytes; the test helper takes raw Vec<u8> and wraps
it (Vec -> Bytes is O(1)).
@mfw78
feat(sdk): log run diagnostics through the tracing facade
752163a 
The run logged through the LoggingHost seam, which the
guest tracing capture cannot observe - module tests proving
behaviour identity for keeper ports assert on tracing events.
Route the submit-path diagnostics through the tracing macros with
the wording the legacy twap poll loop established, and give
ConditionalSource a defaulted label so shared log lines attribute
the strategy that produced them.
@mfw78
refactor(twap-monitor): port the poll loop onto the keeper composition
a56902a 
Reduce strategy.rs to decode and evaluate: log decoding persists
watches through the keeper watch set, and the
getTradeableOrderWithSignature evaluation moves behind
ConditionalSource. The hand-rolled gate keys, watch-update
lifecycle, submitted: markers, and submit-retry dispatch are
deleted in favour of cow::run over the keeper stores and
retry ledger, still on the legacy CowApiHost seam. The MockHost
dispatch tests are untouched - the behaviour-identity proof for
the port.
@mfw78
feat(sdk-test): namespace the mock local store and share its limits
bf4d14e 
Rework MockLocalStore to mirror the runtime store's shape: namespaced
views over one shared row map, so identical key strings in sibling
namespaces never collide, plus store-wide entry and byte limits that
span namespaces the way one database file does. Fault injection stays
per-view.
@mfw78
feat(sdk-test): add the programmable MockVenue on the cow-api seam
02e6276 
Script per-call venue behaviour where MockCowApi replays one response:
a strictly-draining queue of submit outcomes with a configurable
steady-state fallback, per-route response sequences whose final entry
sticks (a terminal order status persists across re-polls), and venue
fault injection that overrides both without consuming them. MockHost
grows a defaulted venue type parameter so the composed host carries
either mock on the same CowApiHost seam. Acceptance tests drive the
keeper run across multi-tick retry, backoff, and outage
scenarios, and module-shaped strategy code against status sequences.
@lgahdl @claude @mfw78
feat(chain): cap JSON-RPC response size before lowering into the guest (
8ec494b 
#154)

Adds a configurable `[limits.chain] response_max_bytes` knob (default
1 MiB) that is enforced host-side in `chain::request()` before the
response string is copied into the guest heap. Oversized responses are
rejected with an `InvalidInput` fault, a `WARN` log, and a dedicated
`shepherd_chain_response_capped_total` metric.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfSsJPYDQ1GQGbnSVFxATx
@lgahdl @claude @mfw78
fix(#154): bound batch aggregates, harden config, test the real wiring
74e6866 
Review follow-ups on the chain response cap:

- request_batch: the per-entry cap bounds each body, but the aggregate
  Vec<RpcResult> lowered into guest memory was unbounded - ~64 entries
  just under the cap is ~64 MiB, exactly the guest-heap saturation
  issue #154 exists to prevent, reachable via the block-range chunking
  the issue itself recommends. A running total now converts entries
  past the cap into typed invalid-input results.
- config: renamed [limits.chain].response_max_bytes to
  response_body_max_bytes (u64) for exact symmetry with the
  [limits.http] sibling; a degenerate 0 saturates to 1, matching the
  logs/poison zero handling.
- new harness test drives the cap through the real path - config ->
  ModuleLimits -> HostState -> chain::request - with the price-alert
  module: an over-cap oracle response surfaces to the guest as the
  typed fault and never reaches classify. The existing unit tests only
  covered the free function, so a deleted and_then would have passed
  the suite.
- TOML parse tests for [limits.chain] (absent -> 1 MiB default,
  override, zero saturation), mirroring the HTTP section's tests.
- engine.example.toml documents the new section; warn message dash
  matches house style.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TfSsJPYDQ1GQGbnSVFxATx
@mfw78 mfw78 force-pushed the feat/154-chain-response-size-cap branch from 52ca638 to 74e6866 Compare July 14, 2026 12:47
@lgahdl lgahdl linked an issue Jul 14, 2026 that may be closed by this pull request
chain: cap response size before lowering into the guest #154
Closed
@mfw78 mfw78 force-pushed the feat/m1-mock-venue branch from 02e6276 to 6076837 Compare July 14, 2026 23:33
Base automatically changed from feat/m1-mock-venue to dev/m1 July 14, 2026 23:36
@mfw78 mfw78 added the base: dev/m1 Open PR targets dev/m1 integration branch — merge AFTER develop PRs; develop merged in last. label Jul 15, 2026
@mfw78

mfw78 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Superseded by #354 — rebased onto current dev/m1 (dropped redundant train commits), conflicts resolved against the landed train, Claude trailers + body footer stripped, authorship preserved (@lgahdl). Closing in favour of #354.

@mfw78 mfw78 closed this Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jean-neiverth jean-neiverth jean-neiverth approved these changes
@mfw78 mfw78 Awaiting requested review from mfw78 mfw78 is a code owner

Assignees

No one assigned

Labels

base: dev/m1 Open PR targets dev/m1 integration branch — merge AFTER develop PRs; develop merged in last. pr-train Part of the current stacked PR train; lands in sequence, do not merge out of order.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

chain: cap response size before lowering into the guest

3 participants

@lgahdl @mfw78 @jean-neiverth