ConnectionRecoveryErrorAsync is never raised when topology recovery fails, contrary to its documentation · Issue #2014 · rabbitmq/rabbitmq-dotnet-client · GitHub
Skip to content

ConnectionRecoveryErrorAsync is never raised when topology recovery fails, contrary to its documentation #2014

Description

@lukebakken

Note

This issue was written by Claude (Anthropic's Claude Code) under the direction of @lukebakken. It was found while writing a regression test for #1995: the test was asserting on this event and passed whether or not the fix was present, which turned out to be because the event never fires for this path. The code paths below were read directly and confirmed with a diagnostic build against a live broker; the suggested directions are AI-drafted starting points, not settled decisions.

IConnection.ConnectionRecoveryErrorAsync documents itself as covering topology recovery:

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/main/projects/RabbitMQ.Client/IConnection.cs#L166-L173

/// <summary>
/// Raised when the connection recovery fails, e.g. because reconnection or topology
/// recovery failed.
/// </summary>

It is raised from exactly one place, and that place only handles reconnection:

Location (on main) Method Behaviour on failure
AutorecoveringConnection.Recovery.cs:416 TryRecoverConnectionDelegateAsync (starts at 383) raises ConnectionRecoveryErrorAsync
AutorecoveringConnection.Recovery.cs:359 TryPerformAutomaticRecoveryAsync (starts at 309) logs "Exception when recovering connection. Will try again after retry interval.", aborts the inner connection, returns false. No event.

Topology recovery runs inside TryPerformAutomaticRecoveryAsync, after reconnection has already succeeded: exchanges, then queues, then bindings, then channels and consumers. Any exception from those, including one that ShouldRetryRecoveryAfter deliberately classifies as retry-worthy, lands in the catch at 359. So the documented "topology recovery failed" case raises nothing at all.

Why it matters

An application that subscribes to ConnectionRecoveryErrorAsync to detect recovery trouble sees nothing while topology recovery fails and silently retries in a loop. The connection reports IsOpen, RecoverySucceededAsync eventually fires if a later attempt gets through, and the only trace of the failed attempts is an ESLog entry. That is precisely the class of silent failure #1993 was about, one level up.

It is also the reason a regression test for #1995 could not observe a failed attempt. Verified with a diagnostic build: the retry decision fired (shouldRetry=True, so the attempt was failed and retried), while a subscriber to ConnectionRecoveryErrorAsync recorded zero events.

Note that #1995 raises the stakes slightly. Now that the retry classification is applied after a custom TopologyRecoveryExceptionHandler runs, more topology failures fail the attempt rather than being swallowed, and every one of them is invisible to this event.

Possible directions, not settled

  1. Raise ConnectionRecoveryErrorAsync from the catch in TryPerformAutomaticRecoveryAsync as well, making the event match its documentation. Applications that already subscribe would start seeing events they never saw before, which is the point, but it is a behaviour change.
  2. Leave the event alone and correct the documentation to say it covers reconnection only. Cheapest, but leaves topology recovery failures unobservable, and there is then no event that covers them.
  3. Add a separate event for topology recovery failure, leaving the existing one's meaning untouched. Most explicit, but new public API and so subject to the 7.3.0 API gate in Ship API changes #1923.

Option 1 looks closest to the documented intent. Whichever is chosen, TopologyRecoveryException already carries the failing entity in its message, so the event argument would be useful without new types.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

A-recoveryArea: Automatic connection and topology recovery.C-bugCategory: This is a bug.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions