Add load shedding for backend message processing

[SimonHeybrock]

To do

Figure out if this strategy is actually necessary - could we instead double the batch length (time), processing more events at a time? I suspect that it might be a good and arguably better improvement, even though it might not be sufficient in the limit of high event rates. Or is that already happening naturally? Need to check how the message batcher behaves.

Summary

Closes #378

When the backend can't keep up with the Kafka message stream, a new LoadShedder selectively drops bulk event data while preserving control messages and f144 logs.

Detection: Consecutive non-None batcher results signal overload — zero-cost, no API changes, directly reflects falling behind. Hysteresis (5 consecutive batches to activate, 3 idle cycles to deactivate) prevents oscillation.

Shedding strategy: Uses exponential levels where each level keeps 1/2^N of droppable messages (level 1 = 50% drop, level 2 = 75%, level 3 = 87.5%). Capped at level 3 — this handles up to 8x overload while still producing usable data (~1-2 messages per source per batch at typical rates). Each escalation/de-escalation step requires the same consecutive-cycle thresholds, so the system ramps up under sustained overload and ramps down gracefully when load subsides. Only bulk data (DETECTOR_EVENTS, MONITOR_EVENTS, MONITOR_COUNTS, AREA_DETECTOR) is eligible — all LIVEDATA_* control streams and LOG (f144) are preserved.

Reporting: Drop statistics use a rolling 60-second window (10 buckets x 6s) so the dashboard shows a meaningful current rate rather than an ever-growing cumulative counter. The dashboard displays an amber "SHEDDING" badge and the drop rate percentage on affected workers, with the rate naturally decaying to 0% after shedding stops. Level transitions are logged at warning severity.

Wiring: OrchestratingProcessor runs shed() before the batcher and report_batch_result() after. Load shedding is enabled by default but can be disabled via enable_load_shedding=False (used in integration tests).

Test plan

• tests/core/load_shedder_test.py — activation, escalation, de-escalation, max level cap, drop rates at each level, rolling window decay, state reporting
• tests/kafka/status_message_test.py — roundtrip for is_shedding, shedding_level, messages_dropped, messages_eligible fields
• Full test suite passes (2484 passed, 28 skipped)
• Manual testing: shedding activates and escalates under sustained load, caps at level 3, de-escalates when load subsides

🤖 Generated with Claude Code

[SimonHeybrock]

Superseded by #1271. The premise here — that we need a shedding controller because the batcher can't absorb overload — no longer holds: AdaptiveMessageBatcher already escalates the batch length with hysteresis, so this adds a second feedback loop on the same signal, each perturbing the other's input. The open question at the top of this description ("could we instead double the batch length (time), processing more events at a time? ... Or is that already happening naturally?") is answered: yes, up to max_level=3.

The failure this was meant to prevent turned out to be an unbounded buffer rather than a missing controller. RateAwareMessageBatcher._overflow grew without limit whenever a service iteration outran the batch length — batch() releases at most one batch length per call while the source hands over a whole iteration's worth — and that is what OOM-killed detector_data on TBL, with the Kafka consumer reporting zero lag throughout because the backlog sits downstream of it.

A bounded buffer sheds exactly the surplus, so it self-tunes to the shortfall; the fixed fractions here (50/75/87.5%) cannot match an arbitrary one, and TBL's 1.41x overload needed roughly 29%. It also inherits selectivity for free: only GATED_STREAM_KINDS reach the overflow, so control, log and context data can never be shed without stating the rule.

Not closing yet: the reporting half — the rolling drop-rate window, the status fields, and the dashboard badge — is still the prior art for the third item of #378 (propagate to the user), which #1271 deliberately leaves open. Worth harvesting that before this is closed.