buzz-acp: 429s are retried with no backoff (median gap 0.0000s, 20-42/sec) — a transient rate limit becomes a self-sustaining storm

[jimparkins]

Summary

When the relay starts returning 429 Too Many Requests, buzz-acp emits
retries with no measurable delay between them — hundreds per second. A
transient rate limit becomes a sustained, self-sustaining storm that the agent
never escapes, and which starves every other client on the same host.

Measurement

Taken from a single agent's log (one buzz-acp, lazy pool, zero agent
child processes running):

429 events sampled : 175
inter-retry gap    : min 0.0000s   median 0.0000s   max 0.65s
gaps under 100ms   : 162 / 171

If there were exponential (or any) backoff, the gaps would trend upward over
the sample. They do not:

first 10 gaps, median : 0.0003s
last  10 gaps, median : 0.0000s

Aggregated per wall-clock second, the same log shows a median of 20 and a
peak of 42 429 responses per second, sustained.

The part that makes it a storm, not just noise

While the agent was producing 20–42 rejected requests per second, a plain
curl to the same relay from the same host returned 8/8 × HTTP 200. So the
relay was perfectly willing to serve this host at a normal request rate — the
agent's own retry volume was what kept it above the limit.

Earlier in the same session, with more agents running, curl got 0/10. Stop
the agents and it returns to 10/10 immediately. The limit is reached by
client behaviour, and the retry loop is what holds it there.

Two possible mechanisms — I can't distinguish them from outside

I want to be careful not to over-claim the cause:

1. No backoff on retry — each rejected request is retried immediately.
2. High in-flight concurrency — many parallel requests each get a 429 and
   each logs, so the log rate reflects fan-out rather than retry rate.

The observable effect is the same and the fix probably overlaps (backoff plus a
concurrency cap on the publish path), but they're different bugs and worth
telling apart from inside the code. The log line is
buzz_acp::relay: POST /events returned retriable HTTP 429 Too Many Requests.

Impact

This is the amplifier behind several other failures, including #5555 (an agent
that never joins a channel it has been added to) — the agent cannot make
progress because it is continuously rejected, and it cannot recover because
nothing slows it down.

It also degrades unrelated clients: with agents running, ordinary requests from
the same machine are rejected too, so a user's own desktop or CLI appears
broken while an agent is in this state.

Suggested fix

• Exponential backoff with jitter on retriable statuses, and honour
  Retry-After when present.
• A ceiling on concurrent in-flight publishes per agent.
• Log rate-limiting for this path — 175 identical lines in a few seconds buries
  every other diagnostic in the terminal, which is how this went unnoticed
  for the first hour.

Context

macOS, Buzz Desktop 0.5.8, agent runtime buzz-agent, deployed through an
out-of-tree buzz-backend-* provider running agents in Orca worktrees.
Relay is self-hosted, Cloudflare-fronted.

[jimparkins]

This appears to be the root cause behind #5555, not just a neighbour of it.

Fresh evidence from a clean-room attempt. I removed every confound I could: all other agents stopped, no competing processes on the host, relay verified healthy immediately before and during.

A single freshly-deployed agent, zero agent child processes, on a relay that answers curl with 8/8 × HTTP 200 at the same moment:

queue depth cap reached — dropped oldest event : 171
POST /events returned retriable HTTP 429       :  39
subscribed to channel                          :   0

So with the relay demonstrably willing to serve this host, the agent still rate-limits itself during startup, overflows its outbound queue 171 times, and never completes channel subscription.

The sequence looks like:

1. agent starts, connects, discovers 0 channels (correct — it has just been created)
2. it is added to a channel in Desktop
3. its outbound publish path (profile/presence sync and the membership follow-up) fires faster than the relay's limit allows
4. the retries have no backoff (this issue), so the limit is never released
5. the outbound queue overflows and drops events
6. subscription never completes — and per buzz-acp: an agent added to a channel during relay rate-limiting never joins it and never recovers — no membership reconciliation after queue overflow #5555 nothing re-reconciles, so it never recovers

If that reading is right, #5555 is a symptom: the agent is not failing to reconcile after a transient loss, it is failing to get through startup at all because it is continuously rejecting itself. Fixing backoff here may resolve both.

Two caveats, because I have already had to retract one confident causal claim on #5555 today:

• I cannot see from outside whether the 429 volume is retry-without-backoff or high in-flight concurrency (as noted in the original report). Either produces this.
• I have not proven step 3-4 causes step 6; I have shown they co-occur under conditions where the relay itself is healthy, which rules out "the relay was down" as the explanation.

Reproduction is simply: deploy a new managed agent, add it to a channel, and watch its log. On this setup it happened on 4 of 4 attempts.