Bunny does not recover from Rabbitmq Broker restart but reconnects on manual restart of service

[akilaragh]

Bunny version:2.14.2
RabbitMQ version: 3.8.17

When broker restarts happen, the connection is not able to recover and the consequent publishes give the following error. When the service is restarted manually, the connection is fine and it is able to publish.

Trying to send frame through a closed connection. Frame is [#\u003cAMQ::Protocol::MethodFrame:0x000000012bd5fdc8 @payload=\"\\x00\u003c\\x00(\\x00\\x00\\x06\\x10private@@members\\x00\", @channel=1\u003e, #\u003cAMQ::Protocol::HeaderFrame:0x000000012bd5f2b0 @payload=\"\\x00\u003c\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x000\\x98\\x00\\x18application/octet-stream\\x02\\x00\", @channel=1\u003e, #\u003cAMQ::Protocol::BodyFrame:0x000000012bd5f260 @payload=\"{\\\"type\\\":\\\"update\\\",\\\"id\\\":114161848,\\\"attributes\\\":{}}\", @channel=1\u003e] Trying to send frame through a closed connection. Frame is [#\u003cAMQ::Protocol::MethodFrame:0x000000012bd5fdc8 @payload=\"\\x00\u003c\\x00(\\x00\\x00\\x06\\x10private@@members\\x00\", @channel=1\u003e

[michaelklishin]

Trying to send frame through a closed connection

is not evidence of Bunny not trying to reconnect. It might be trying to do that. It's an exception
you would get when you attempt to publish on a closed connection. Bunny won't swallow those or try to
handle them for you.

[akilaragh]

@michaelklishin The connection and publish seem fine when the service is restarted manually. Also only some of the services connected to the RabbitMQ broker face this issue. Other services seem to reconnect with the same broker without an issue

[michaelklishin]

It's a matter of timing. When you restart RabbitMQ, your app keeps trying to republish and runs into this. When you restart the service, it won't publish initially. Or something like that. It doesn't
change the meaning of the exception, nor does it relieve you from the
responsibility for handling such failed publishers and retrying.

You can use Bunny::Connection#open? and Bunny::Channel#open? to determine their state.

[akilaragh]

So when this issue happened the service was not able to publish for more than 30 mins post which we did a restart and publish worked instantly. When you say we have the responsibility of handling failures, do you mean we would have to handle opening a new connection and closing the old one ? We tried that and it caused a whole new set of issues like, the
1.old connection not getting closed properly- Timeout when connection close was called
2.Publish was stuck in the new connection, since publish is async no errors were thrown but the publish also didn't happen.

[Schmitze333]

Hi 👋 .

I hope it's ok that I kind of hijack this discussion, but it fits since our RabbitMQ cluster has been in a similar state lately:
A cluster node was replaced and a fair amount of applications were not able to reconnect although the cluster was healthy again.
As above, a restart of the applications solved the issue.

Other than above, our the applications were consumers (backed by the ears gem).

We tried to circumvent such situations in the future by utilising the Bunny::Session option recovery_attempts, but failed.
I found a closed, old issue that perfectly describes the reason why we failed.

Unfortunately, the issue was closed with a PR that did not add the desired behaviour of the OP.

I'd like to discuss the possibility of adding a recovery_failed option to the Bunny session.
It would receive a Proc (like also recovery_completed and recovery_attempt_started do) and would be called when the number of recovery_attempts is reached and the connection could still not be recovered.

I'd happily provide a according PR!

[Schmitze333]

Why not investigate what exactly prevents it from succeeding or retrying in your environment?

I agree that this would be the very, very best thing to do.
And let me put this in the right context: We have a huge cluster (~5k connections) and in 99.99% of all cases swapping/restarting nodes works perfectly fine. But we had a situation where it did not work for all apps two weeks ago. This was due to an unhealthy cluster state. Unfortunately, we were not able to save all the log files from all cluster nodes and hence are now unable to fully understand the root cause for what had happened and the reason for the reconnect issues.

At the moment management is pressing...

And what will this new proc do then?

Well, Bunny would just call the Proc in case all recovery attempts failed. The user could put in whatever he desires (send a notification, etc.). In our case I would let the Proc just throw an error, causing the app to crash and Kubernetes to restart the pod.

I'd consider a PR that makes recovery retries more resilient...

IMO the recovery retries work perfectly fine.

#549 was about recovery attempt limit not kicking in.

Not sure. I read it like "Recovery attempt limit works, but has no effect for the client."

The closing PR solves it in a way that an error would be raised whenever one uses the connection or channel in the situation all attempts have failed. Unfortunately, this does not work in our case since we are not publishing but consuming.
Of course, we could implement some supervisor thread in our consumer framework checking the connection state, but a feedback mechanism for expired recovery attempts would allow a much more elegant solution.

I'm suggesting a callback since:

• It goes in line with the design choice to have callbacks around the recovery mechanism.
• Would be the minimal invasive solution to the problem.
  Raising an exception (as suggested in the issue) would change the current behaviour unless another option raise_on_failed_recovery_attempt (or the like) is introduced.

[womblep]

I am hitting a similar problem and in looking at other peoples problems similar to mine and the OP, it looks like it is happening to clusters. Potentially in environments like kubernetes or in my case AWS fargate.
My suspicion is there is a logic gap when recovering the connection to a rabbitmq node via a load balancer and the node is not completely up yet. Some combination of factors setting a flag which makes it look like the connection is up but it isnt.

@michaelklishin can you release a new gem to rubygems with the network recovery changes you merged in feb? #658 merged into 2.21.0 milestone. I made the changes because in some cases the reconnect ion was stalling with openssl issues and this issue maybe linked with that.

[Schmitze333]

@womblep, thank you for answering!

In order to provide one more data point for your investigation:

A RabbitMQ cluster on our side consists of 5 AWS EC2 instances running Ubuntu 22.04 behind a load balancer.
Before a node is added to the load balancer two consecutive health checks against port 5672 must be passed with an interval time of 45 seconds. So, we are pretty sure that the node is up-and-running before any application can connect to it.

But we never payed attention to the cluster formation process.
What would happen when a node is reachable but hasn't joined the cluster, yet. It accepts a connection and then afterwards joins the cluster. What happens to this connection? Could this lead to the problem?

[womblep]

@Schmitze333 Michael has very kindly updated the gem with the pull request that handles more error conditions for network recoverability. You can even add to or change the list using the new accessor "recoverable_exceptions".
Maybe turn verbose on and see what condition is causing the connection to stall for you, then you can add it to the list. StandardError is on the list so you might find it works for you already.
I was finding that in some edge cases, openssl was failing but the exception wasn't being picked up as a recoverable error. Now it should be, assuming my testing goes well :-)

[Schmitze333]

Yeah cool!

🤞This prevents the problem entirely.
But if not, with the release of 2.22.0 we have the tooling at hand to at least mitigate this situation.