Took a glance at consumer recovery after a .NET client fix: two possible items

[lukebakken]

Note

This post was written by Claude (Anthropic's Claude Code) under the direction of @lukebakken, who asked it to look at the Java client after we finished related work on the .NET client. The source reading below was done against main; no Java repro was run, so the reachability argument is timing analysis rather than measurement.

We just fixed a topology recovery bug in the .NET client (rabbitmq/rabbitmq-dotnet-client#1993): when the basic.consume issued during consumer recovery exceeded the RPC timeout, recovery swallowed the failure and reported success, while the broker had already registered the consumer. Deliveries then went to the fallback consumer and were dropped unacked, so the queue silently stopped being consumed on a connection that still reported open.

I directed Claude to take a glance at the Java client to see whether anything similar lurks here. The .NET bug as reported does not transfer - the two clients diverge in the places that matter. But two things came up that might be worth a look. Framing this as "we took a glance and maybe found these" rather than as bug reports, since neither has a reproducer and you know this code far better than we do.

First, why the .NET bug does not apply. Java has no exception-type mismatch: privateRpc wraps the timeout into ChannelContinuationTimeoutException, and recovery retry conditions are user-supplied predicates rather than a type list. More decisively, an unknown consumer tag is loud rather than silent - processDelivery throws IllegalStateException when no defaultConsumer is set, which propagates to the main loop and tears the connection down for recovery. The comment there citing "bug 22587" reads like this was a deliberate choice, and it is what contains the blast radius.

Possible item 1: a discarded stale RPC reply is silent. After a timeout, wrapTimeoutException calls cleanRpcChannelState, which nulls _activeRpc. If the basic.consume-ok arrives afterwards, handleCompleteInboundCommand finds no outstanding RPC and drops it with no log line. Since _consumers.put happens inside transformReply, the tag is never recorded, and Basic.ConsumeOk is not intercepted by processAsync so nothing else records it either. Net effect: broker-side consumer exists, client-side tag absent, and the next delivery kills the connection via the IllegalStateException above. Self-correcting and visible, unlike the .NET case, but a debug or warning log on the discard might have made #708 easier to diagnose. Worth noting the misrouting guard is gated on channelShouldCheckRpcResponseType, which defaults to false.

Possible item 2: no prebuilt retry condition covers a timeout. topologyRecoveryRetryHandler has no initializer, so retry is opt-in. When it is enabled, the only prebuilt condition is CHANNEL_CLOSED_NOT_FOUND, a 404 check that cannot match ChannelContinuationTimeoutException. A ready-made condition for timeouts would be the direct analogue of what we did in .NET. Separately, beginAutomaticRecovery calls notifyRecoveryListenersComplete() unconditionally after recoverTopology, so a per-entity failure still surfaces as a successful recovery to listeners.

On reachability, which is why we are not filing these as bugs: DEFAULT_CHANNEL_RPC_TIMEOUT is 10 minutes, while setHeartbeat sets the socket timeout to a quarter of the heartbeat and handleSocketTimeout throws MissedHeartbeatException after 8 misses. At the default 60s heartbeat that is roughly 2 minutes of silence, so the heartbeat should kill the connection well before the RPC timeout fires and recovery starts over cleanly. The window seems to require a short explicit channelRpcTimeout or disabled heartbeats.

Questions, if any of this is worth your time: is the silent discard intentional, or would a log line there be welcome? And is a timeout-based retry condition something you would want in TopologyRecoveryRetryLogic, or is the current opt-in design deliberate? Happy to be told this is all a non-issue given the heartbeat interaction - we only looked at consumers, and the other entity types are idempotent redeclares where a lost reply matters much less.