-
Notifications
You must be signed in to change notification settings - Fork 563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Keydb server not accepting connections after a few hours #830
Comments
maybe you can use wireshark or tcpdump to inspect what is happening on the network traffic? |
Trying to deal with same issue a long time. |
Thanks for your response. |
Then you probably need to diagnose with Wireshark/tcpdump. |
Thanks @keithchew and @rez0n . For me it is not a diskspace or memory issue. i have plenty even when the issue exists. |
I'm again faced with the issue.
It was stuck 4 hours ago and nothing more happens. |
I have not tried tcpdump yet, but the keydb logs is exactly same for my issue as well. No errors and suddenly freezes out and finally stops accepting further connections. |
Some other ideas:
|
We just experienced the same issue. The system had low load,memory, and disk i/o usage. Keydb memory usage and connections were low and nothing in the log. It just stopped responding and accepting connections. |
Is the process still running? Another one I encountered a while back, not sure if any of you hit this one: Although I caught it when doing a scan from a module, it does not appear to be isolated to that scenario. |
If the process died (instead of alive and not accepting connections), then I would also recommend searching for known crash related issues, and see you have encountered one of them. A reference to a list of issues (most crash related) I discovered while testing is below: https://github.com/Snapchat/KeyDB/issues/created_by/keithchew |
We encounter the same problem currently. Started with a fresh debian 12 based install, running on a proxmox based vm, using the ppa packages. We are running a 3 node cluster and it will work for 15-60 minutes without issue, and then one node just goes into a state that matches what was described above:
To rectify this, we have to kill -kill the stuck process. It was mostly but not exclusively the first node in the cluster to get stuck. Probably specific to proxmox: We had to change the cpu type from kvm64 to host and enable +aes to get it working at all, before that the keydb-server would not start - but strace showed something about checking for cpu flags as one of the last few entries.
|
I have a set of applications connecting to a Keydb server. If i restart my services multiple times the keydb server becomes unresponsive. It stops accepting further connections, but keydb will still be running though. Once it becomes unresponsive it cant accept connections even from within the keydb machine. This makes me believe that i can eliminate any proxy/connectivity issues.
Therefore my suspicions were primarily based on the keydb machine's resource utilization and the number of active connections the keydb deals with.
The resource utilizations are minimal with a very low memory/CPU usage when the keydb is unresponsive. There is enough disk space in the machine.
The connection count also remains pretty much the same when the connectivity is successfull and unsuccessfull. Sometimes during the unresponsive state it has much lesser active connections compared to when the connections are successfull.
The keydb server logs stop at the moment the connections are blocked, no logs after that. I have enabled debug logs also to track down the issue. The server machine logs also shows no error. It shows keydb logs normally and suddenly starts showing connection refused logs.
Any help on what might be the issue or what i am missing here would be great.
The text was updated successfully, but these errors were encountered: