buzz-acp: an agent added to a channel during relay rate-limiting never joins it and never recovers — no membership reconciliation after queue overflow

[jimparkins]

Summary

An agent added to a channel while the relay is rate-limiting never joins it,
and never recovers — it stays at zero channel subscriptions indefinitely,
long after the relay is healthy again. There is no periodic reconciliation of
channel membership, so a single overflow window strands the agent permanently.

What I observed

A freshly deployed managed agent started, connected, and reported:

INFO buzz_acp: connected to relay at wss://…
INFO buzz_acp: discovered 0 channel(s)

Zero channels is correct at that moment — it had not been added to one yet.

I then added it to a channel through Buzz Desktop (agent profile → Channels →
Add to channel, role bot). The dialog succeeded.

Meanwhile the relay began rate-limiting this host hard. Over the next few
minutes the agent's log accumulated:

Log line	Count
POST /events returned retriable HTTP 429 Too Many Requests	172
queue depth cap reached — dropped oldest event	78
POST /events network error: error sending request	12

The rate limiting was not specific to this agent — a plain
curl https://<relay>/ from the same host returned HTTP 429 at the same
time, so it was an IP-level limit affecting every client on the machine.

The relay recovered (curl → HTTP 200). Ninety minutes later the agent
still had zero channel subscriptions — no subscribed to channel … line ever
appeared, and it could not be mentioned.

Why this is more than "the relay was busy"

The transient failure is expected and fine. The problem is that the recovery
path doesn't exist:

• buzz-acp subscribes to membership notifications once at startup and reacts
  to them as they arrive.
• Channel discovery runs once, at startup.
• When the inbound queue overflows (queue depth cap reached), whatever was
  dropped is simply gone.
• Nothing re-runs discovery or re-reconciles membership afterwards — not on
  queue overflow, not on reconnect, not on a timer.

So the window in which a membership notification can be lost is small, but the
consequence is permanent and silent: an agent that looks completely healthy
(process alive, relay connected, green dot in the UI) and is simply not in any
channel.

Impact

Worse for provider-backed agents, which cannot be restarted. Protocol v1
has no undeploy, Stop and Start are no-ops for them (see #4605), so the
documented recovery — restart the agent — is unavailable. The only remedy is
delete and redeploy.

Suggested fix

Any of these would close it:

1. Re-run channel discovery after a queue overflow. The queue depth cap reached path already knows something was dropped; treat that as "my view of
   membership may be stale".
2. Periodic membership reconciliation — cheap, and self-heals every cause of
   drift, not just this one.
3. Re-discover on reconnect, since a reconnect already implies a gap.

Related

• buzz-acp: dynamic subscription replays from the membership notification timestamp — messages sent before the notification is processed are silently dropped #4564 — also about membership notifications, but the opposite direction: the
  notification is processed and the replay floor drops older messages. Here
  the subscription never happens at all.
• Provider-backed managed agents have no management surface on the owner's other desktops — sync agent records across devices #4605 — why the usual "restart it" workaround is unavailable for
  provider-backed agents.

Observed on Buzz Desktop 0.5.8 (macOS), agent runtime buzz-agent, deployed
through a third-party buzz-backend-* provider.

[jimparkins]

Correction to my own report — the "relay had recovered" claim is not safe.

When I filed this I checked the relay with a single curl that returned HTTP 200, and concluded the rate limiting had passed and the agent had failed to re-reconcile afterwards. That inference was too quick.

Re-testing properly, the relay is intermittently limiting this host rather than having recovered:

try1 HTTP 200
try2 HTTP 429
try3 HTTP 429

DNS resolves, TCP 443 connects, and unrelated hosts return 200, so it is not local connectivity — it is a per-IP limit that a single sample can easily catch on a good roll.

What this means for the report:

• The observable symptom stands: an agent added to a channel never subscribed, and stayed at zero subscriptions for 90+ minutes while Buzz Desktop showed Channels 1 for the same agent. Membership and subscription genuinely diverged.
• But I cannot currently distinguish between (a) the membership notification being lost to queue overflow with no reconciliation path, and (b) the agent simply being unable to reach the relay for the whole period. I asserted (a); (b) is not excluded.

I reproduced the same end state a second time on a fresh agent — added to the channel while a curl showed HTTP 200, still zero subscriptions — but given the intermittency above, that 200 does not prove the relay was usable at the moment the notification would have arrived.

So: treat the reconciliation question as open rather than demonstrated. The narrower and still-true observation is that there is no visible recovery path — no log line, no retry, no re-discovery, and (for provider-backed agents, per #4605) no way to restart the process to force one. Whichever cause applies, the agent gives no signal that it is not in a channel it appears to be in.

Apologies for the noise; I would rather correct this than leave a confident but unproven cause attached to the issue.