Relay Server: Not Enough Memory on Health Check even though stats show otherwise #3330
Comments
|
Duplicate of #3327 |
|
I think it's a memory leak. |
|
What I can confirm is the fact, that doing the upgrade to 24.9.0 did NOT fix the issue. error in the docker compose logs. |
|
What does the event volume look like for you? Did this start happening after upgrading to 24.8.0? |
|
We did the 24.8.0 Sentry update on the 1st of September. As you can see there are sections where it works fine but then sometimes for a few hours, sometimes even for days there are no events being processed.
|
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
getsantry
bot
moved this from Waiting for: Community
to Waiting for: Product Owner
in GitHub Issues with 👀 3
|
Could you track your RAM/CPU usage as well? Wondering if there is a correlation there. |
|
I can also see errors related to getsentry/snuba#5707 in my logs |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
getsantry
bot
moved this from Waiting for: Community
to Waiting for: Product Owner
in GitHub Issues with 👀 3
|
I will have to try and get some system metric info stats running to give you the requested info |
|
Hey @LordSimal can you try this: On your relay:
upstream: "http://web:9000/"
host: 0.0.0.0
port: 3000
logging:
level: WARN
processing:
enabled: true
kafka_config:
- {name: "bootstrap.servers", value: "kafka:9092"}
- {name: "message.max.bytes", value: 50000000} # 50MB
redis: redis://redis:6379
geoip_path: "/geoip/GeoLite2-City.mmdb"
health:
max_memory_percent: 1.0Then do Thanks to @Dav1dde |
|
Seems like it panicked |
getsantry
bot
added
Waiting for: Product Owner
and removed
Waiting for: Community
labels
getsantry
bot
moved this from Waiting for: Community
to Waiting for: Product Owner
in GitHub Issues with 👀 3
|
Today 6AM (10h ago), events have stopped coming in again. Here again just for consistency the logs of all containers from the last 12h. I restarted the kafka container via |
|
But what DOES fix the problem is just simply restarting all containers via this of course won't help you understand the root cause of this problem but I don't know what other information I can provide to debug this problem. |
|
@LordSimal I saw these on the logs: Since Found this issue, but this is specifically for Windows: docker/for-win#8861 |
|
We use Debian 12 with Docker with this official docker repository |
|
Something is definitely wrong with the docker internal network. Even doing a simple I had to restart the whole docker service via |
We have the same problem. Restarting the docker daemon helps. But this is not a good solution right now.
|
|
Just to be extra sure its not a RAM issue - we just upgraded from 32GB to 64GB but this problem still persists. Sentry just always takes half of the available RAM as a base, no matter how much you have.
But looking at the previous comments this indeed seems like a docker internal network problem. |
|
Hey @LordSimal, did you configure a low swappiness setting? I encountered a similar issue running Sentry on a virtualized machine when the swappiness was set to 10. Changing it back to the default value (60) resolved the problem for now—though I’ll continue monitoring it. |
|
|
Just to inform you: Sentry doesn't receive messages after 1-2 days - regularly, therefore I just added a crontab to automatically restart all containers at 02:00 in the morning to prevent this kind of issue from happening. Will try to disable that automatic restart after the next self-hosted update, whenever that releases. |
|
So we have been using 24.10.0 for 1 week now and it seems to be more stable. I think we restarted the containers once manually due to no events being processed anymore but its not as frequent as it used to be. I'll close this issue since it didn't seem to be that widespread, so it may be something related to our setup/network. Thanks to all participating and trying to find this weird bug 👍🏻 hopefully it doesn't return anytime soon. |
|
I actually experience the same issues (even using 24.10.0). Setting fixes it. But I guess that's more of a workaround than a real solution... |
Self-Hosted Version
24.8.0
CPU Architecture
x86_64
Docker Version
27.2.1
Docker Compose Version
2.29.2
Steps to Reproduce
Can't really tell how to reproduce, since it just happens out of nowhere.
Expected Result
Sentry receives errors again
Actual Result
Sentry stops receiving errors after 1-2 days of normal usage.
Checking the docker logs there are a lot of these entries present:
but checking either htop we have enough RAM
as well as checking docker container stats there are no containers using > 95% RAM
docker_compose_logs.txt
latest_install_logs.txt
Event ID
No response
The text was updated successfully, but these errors were encountered: