Replies: 3 comments 3 replies
-
|
@milgner if there were any false positives around delivery acknowledgement timeouts, Team RabbitMQ would be flooded with complaints but that is not the case. Most likely you have N consumers and some were making progress but others did not. The channel was closed and one way or another, this resulted in a double ack. Neither Bunny nor RabbitMQ can do much about this. This is developer's responsibility to handle such timeouts and double acks. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having a hard time putting together the logs of the Ruby application with the ones from the server. There is only one consumer in this Ruby application and nothing in its logs suggests that a timeout happened or something went wrong at all. Finally, even if there is a timeout, I don't understand why it would produce |
Beta Was this translation helpful? Give feedback.
-
|
When a consumer timeout hits, Bunny treats it like any other channel exception.
means that there was no channel instance when a connection had a delivery to dispatch. That's as much as I can tell from one stack trace alone. You are welcome to investigate how this can be reproduced and take a traffic capture if you can. You don't have to wait for 30 minutes at a time, just lower the timeout and if something in a client library like Bunny causes it, it should hit 10 seconds later vs. 30 minutes by default. |
Beta Was this translation helpful? Give feedback.
-
|
I can report that I'm seeing a similar stack trace: This is bunny 2.23.0 and rabbit 3.13.3. This is happening in the midst of many delivery acknowledgement timeouts. In most cases, the timeout causes the server to close the channel, and it works out (mostly) fine. But in some rare cases, we get this stack trace instead. My guess is that there's some sort of RabbitMQ race condition where it tries to deliver a message on this channel at the same time as it's closing it due to the exception, and the client ends up receiving the delivery after it's processed the server-initiated channel close. I realize that this is pretty unlikely claim, and that I don't have enough evidence to back it up. Take it for the hunch that it is. Putting that aside for now, there was one other mystery that stumped me for quite a while. That stack trace came from a I couldn't reconcile this with what I saw in reader_loop.rb: rescue Exception => e
break if terminate?
if !(@session.closing? || @session.closed?)
log_exception(e)
@network_is_down = true
@session_error_handler.raise(Bunny::NetworkFailure.new("caught an unexpected exception in the network loop: #{e.message}", e))
endI would have expected Eventually I figured out that rescue Exception => ein this context actually means rescue ::Bunny::Exception => eI can't say if this behavior was intentional or not. It certainly caught me by surprise. It does explain why this exception was able to kill the thread. |
Beta Was this translation helpful? Give feedback.
-
|
Ruby's exception hierarchy and module namespacing certainly have their fair share of surprises. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
Bunny version: 2.22.0
RabbitMQ version: 3.12.4-1.el8
Recent RabbitMQ log file contents:
Full exception stack traces:
I still have to find out where the timeout in the server log comes from: the application was running fine and all messages had been processed up to that point.
Beta Was this translation helpful? Give feedback.
All reactions