Snuba consumes high cpu #5825

samir34ml · 2024-03-20T12:41:41Z

Self-Hosted Version

24.3.0

CPU Architecture

x86_x64

Docker Version

25.0.4

Docker Compose Version

2.24.7

Steps to Reproduce

I have installed sentry self hosted on an aws EC2. The only different that I have made is the following.
In the process of creating the EC2, I attached a new HDD disk to the server and mounted it. I used HDD disk to mount 2 volumes sentry-kafka and sentry-postgres. I edited the following file install/create-docker-volumes.sh (attached to the issue).
create-docker-volumes.txt
So, now I can mount all the docker volumes data and storage of both sentry-kafka and sentry-postgres to the HDD disk.

The reason why I did this is to save some money and use huge disk of HDD instead of SSD because these volumes consume huge amount of disk space.

Expected Result

normal behavior and events receiving and issue monitoring.

Actual Result

no events no issues high cpu utilization even after uninstalling sentry completely and installing it again.

It worked for a while around day or day and half, then sentry stops to report. No new events or issues or transactions.
I had to reset it and re-install it from the beginning but now even before adding any project cpu is very high above 90%.

I tried to add some logs from the docker compose logs command output.
logs.txt

So any idea regrading what to do next?

Thanks,

Event ID

No response

hubertdeng123 · 2024-03-21T20:13:31Z

Thanks for writing in. Could you answer these questions before proceeding?

What is your even throughput into your instance?
How much CPU are you using?

samir34ml · 2024-03-24T03:32:21Z

Hello @hubertdeng123

for question 1, I hope the attached screenshot answers your question.

And for question 2, I am using t3a.xlarge as EC2 from aws, which has 4 vCPU and 16 GB of memory.

Thanks,

csvan · 2024-03-25T14:12:29Z

As somebody who also deploys Sentry on AWS, I would strongly advice against using t3a instances. Their baseline performance is too low, and the burst performance does not scale well with how Sentry operates.

Look into using m7i instances, at least, or m7i-flex if minimising costs is a concern.

chris-laack · 2024-03-26T07:15:02Z

we use r6a.large instances in the aws. They work very well for us.

levani · 2024-03-28T10:54:23Z

Having the same issue on digitalocean droplet after upgrading to 24.3.0. CPU is constantly at 100%

azaslavsky · 2024-03-29T20:34:33Z

Can you see if the solution suggested here helps in this case as well?

getsantry · 2024-04-20T07:01:34Z

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

rihards-simanovics · 2024-04-24T11:39:31Z

hey just updated to 24.4.1, my CPU is constantly pegged to 100%, snuba seems to be the cause.

azaslavsky · 2024-04-25T19:39:47Z

Can you try the advice posted here?

rihards-simanovics · 2024-04-26T08:14:01Z

Can you try the advice posted here?

hmm actually runs worse without the rust, can see an extra 20% on top for the CPU utilisation from both sentry and arrouns 5% for snuba. have run the docker compose logs -f and it does look like everything is running as expected other than periodical tracebacks from sentry-self-hosted-post-process-forwarder-errors-1, sentry-self-hosted-metrics-consumer-1 and sentry-self-hosted-generic-metrics-consumer-1, in case it helps here are the specific logs:

sentry-self-hosted-post-process-forwarder-errors-1 traces

sentry-self-hosted-post-process-forwarder-errors-1                 | Traceback (most recent call last):
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-post-process-forwarder-errors-1                 |     sys.exit(main())
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
sentry-self-hosted-post-process-forwarder-errors-1                 |     func(**kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-post-process-forwarder-errors-1                 |     return self.main(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-post-process-forwarder-errors-1                 |     rv = self.invoke(ctx)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-errors-1                 |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-post-process-forwarder-errors-1                 |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-post-process-forwarder-errors-1                 |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-post-process-forwarder-errors-1                 |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-post-process-forwarder-errors-1                 |     return __callback(*args, **kwargs)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer
sentry-self-hosted-post-process-forwarder-errors-1                 |     run_processor_with_signals(processor)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals
sentry-self-hosted-post-process-forwarder-errors-1                 |     processor.run()
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-post-process-forwarder-errors-1                 |     self._run_once()
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-post-process-forwarder-errors-1                 |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/sentry/consumers/synchronized.py", line 245, in poll
sentry-self-hosted-post-process-forwarder-errors-1                 |     message = self.__consumer.poll(timeout)
sentry-self-hosted-post-process-forwarder-errors-1                 |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 414, in poll
sentry-self-hosted-post-process-forwarder-errors-1                 |     raise OffsetOutOfRange(str(error))
sentry-self-hosted-post-process-forwarder-errors-1                 | arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}

sentry-self-hosted-metrics-consumer-1 traces

sentry-self-hosted-metrics-consumer-1                              | 07:11:58 [ERROR] arroyo.processing.processor: Caught exception, shutting down...
sentry-self-hosted-metrics-consumer-1                              | 07:11:58 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f46df62f220>...
sentry-self-hosted-metrics-consumer-1                              | 07:11:58 [INFO] arroyo.processing.processor: Processor terminated
sentry-self-hosted-metrics-consumer-1                              | Traceback (most recent call last):
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-metrics-consumer-1                              |     sys.exit(main())
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
sentry-self-hosted-metrics-consumer-1                              |     func(**kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-metrics-consumer-1                              |     return self.main(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-metrics-consumer-1                              |     rv = self.invoke(ctx)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-metrics-consumer-1                              |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-metrics-consumer-1                              |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer
sentry-self-hosted-metrics-consumer-1                              |     run_processor_with_signals(processor)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals
sentry-self-hosted-metrics-consumer-1                              |     processor.run()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-metrics-consumer-1                              |     self._run_once()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-metrics-consumer-1                              |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-metrics-consumer-1                              |     raise ConsumerError(str(error))
sentry-self-hosted-metrics-consumer-1                              | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-metrics-consumer-1                              | Updating certificates in /etc/ssl/certs...
sentry-self-hosted-metrics-consumer-1                              | 0 added, 0 removed; done.
sentry-self-hosted-metrics-consumer-1                              | Running hooks in /etc/ca-certificates/update.d...
sentry-self-hosted-metrics-consumer-1                              | done.
sentry-self-hosted-metrics-consumer-1                              | Traceback (most recent call last):
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-metrics-consumer-1                              |     self._run_once()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-metrics-consumer-1                              |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-metrics-consumer-1                              |     raise ConsumerError(str(error))
sentry-self-hosted-metrics-consumer-1                              | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-metrics-consumer-1                              | 07:12:36 [ERROR] arroyo.processing.processor: Caught exception, shutting down...
sentry-self-hosted-metrics-consumer-1                              | 07:12:36 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f099ef585b0>...
sentry-self-hosted-metrics-consumer-1                              | 07:12:36 [INFO] arroyo.processing.processor: Processor terminated
sentry-self-hosted-metrics-consumer-1                              | Traceback (most recent call last):
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-metrics-consumer-1                              |     sys.exit(main())
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
sentry-self-hosted-metrics-consumer-1                              |     func(**kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-metrics-consumer-1                              |     return self.main(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-metrics-consumer-1                              |     rv = self.invoke(ctx)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-metrics-consumer-1                              |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-metrics-consumer-1                              |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-metrics-consumer-1                              |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-metrics-consumer-1                              |     return __callback(*args, **kwargs)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer
sentry-self-hosted-metrics-consumer-1                              |     run_processor_with_signals(processor)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals
sentry-self-hosted-metrics-consumer-1                              |     processor.run()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-metrics-consumer-1                              |     self._run_once()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-metrics-consumer-1                              |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-metrics-consumer-1                              |     raise ConsumerError(str(error))
sentry-self-hosted-metrics-consumer-1                              | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-metrics-consumer-1                              | Updating certificates in /etc/ssl/certs...
sentry-self-hosted-metrics-consumer-1                              | 0 added, 0 removed; done.
sentry-self-hosted-metrics-consumer-1                              | Running hooks in /etc/ca-certificates/update.d...
sentry-self-hosted-metrics-consumer-1                              | done.
sentry-self-hosted-metrics-consumer-1                              | Traceback (most recent call last):
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-metrics-consumer-1                              |     self._run_once()
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-metrics-consumer-1                              |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-metrics-consumer-1                              |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-metrics-consumer-1                              |     raise ConsumerError(str(error))
sentry-self-hosted-metrics-consumer-1                              | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-metrics-consumer-1                              | 07:13:14 [ERROR] arroyo.processing.processor: Caught exception, shutting down...

sentry-self-hosted-generic-metrics-consumer-1 traces

sentry-self-hosted-generic-metrics-consumer-1                      | 07:59:24 [ERROR] arroyo.processing.processor: Caught exception, shutting down...
sentry-self-hosted-generic-metrics-consumer-1                      | 07:59:24 [INFO] arroyo.processing.processor: Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7fc73d1035e0>...
sentry-self-hosted-generic-metrics-consumer-1                      | 07:59:24 [INFO] arroyo.processing.processor: Processor terminated
sentry-self-hosted-generic-metrics-consumer-1                      | Traceback (most recent call last):
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/bin/sentry", line 8, in <module>
sentry-self-hosted-generic-metrics-consumer-1                      |     sys.exit(main())
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/__init__.py", line 193, in main
sentry-self-hosted-generic-metrics-consumer-1                      |     func(**kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
sentry-self-hosted-generic-metrics-consumer-1                      |     return self.main(*args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
sentry-self-hosted-generic-metrics-consumer-1                      |     rv = self.invoke(ctx)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return __callback(*args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-generic-metrics-consumer-1                      |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 69, in inner
sentry-self-hosted-generic-metrics-consumer-1                      |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return __callback(*args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
sentry-self-hosted-generic-metrics-consumer-1                      |     return f(get_current_context(), *args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/decorators.py", line 29, in inner
sentry-self-hosted-generic-metrics-consumer-1                      |     return ctx.invoke(f, *args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
sentry-self-hosted-generic-metrics-consumer-1                      |     return __callback(*args, **kwargs)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/sentry/runner/commands/run.py", line 730, in basic_consumer
sentry-self-hosted-generic-metrics-consumer-1                      |     run_processor_with_signals(processor)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/sentry/utils/kafka.py", line 13, in run_processor_with_signals
sentry-self-hosted-generic-metrics-consumer-1                      |     processor.run()
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-generic-metrics-consumer-1                      |     self._run_once()
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-generic-metrics-consumer-1                      |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-generic-metrics-consumer-1                      |     raise ConsumerError(str(error))
sentry-self-hosted-generic-metrics-consumer-1                      | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-performance-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-generic-metrics-consumer-1                      | Updating certificates in /etc/ssl/certs...
sentry-self-hosted-generic-metrics-consumer-1                      | 0 added, 0 removed; done.
sentry-self-hosted-generic-metrics-consumer-1                      | Running hooks in /etc/ca-certificates/update.d...
sentry-self-hosted-generic-metrics-consumer-1                      | done.
sentry-self-hosted-generic-metrics-consumer-1                      | Traceback (most recent call last):
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 288, in run
sentry-self-hosted-generic-metrics-consumer-1                      |     self._run_once()
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 355, in _run_once
sentry-self-hosted-generic-metrics-consumer-1                      |     self.__message = self.__consumer.poll(timeout=1.0)
sentry-self-hosted-generic-metrics-consumer-1                      |   File "/usr/local/lib/python3.8/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
sentry-self-hosted-generic-metrics-consumer-1                      |     raise ConsumerError(str(error))
sentry-self-hosted-generic-metrics-consumer-1                      | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: ingest-performance-metrics: Broker: Unknown topic or partition"}
sentry-self-hosted-generic-metrics-consumer-1                      | 07:59:53 [ERROR] arroyo.processing.processor: Caught exception, shutting down...

azaslavsky · 2024-04-26T17:30:36Z

Moved this over to the snuba repo for more domain-specific triage.

FrankHeijden · 2024-04-30T10:42:21Z

Related: getsentry/self-hosted#2931 (comment)

getsantry bot added the Waiting for: Product Owner label Mar 20, 2024

hubertdeng123 added the Waiting for: Community label Mar 21, 2024

getsantry bot removed Waiting for: Product Owner labels Mar 21, 2024

getsantry bot added Waiting for: Product Owner and removed Waiting for: Community labels Mar 24, 2024

azaslavsky added Waiting for: Community and removed Waiting for: Product Owner labels Mar 25, 2024

getsantry bot added Waiting for: Product Owner and removed Waiting for: Community labels Mar 26, 2024

azaslavsky removed the Waiting for: Product Owner label Mar 27, 2024

getsantry bot added the Waiting for: Product Owner label Mar 28, 2024

getsantry bot removed the Waiting for: Product Owner label Mar 29, 2024

azaslavsky added the Waiting for: Community label Mar 29, 2024

getsantry bot added the Stale label Apr 20, 2024

getsantry bot added Waiting for: Product Owner and removed Waiting for: Community Stale labels Apr 24, 2024

getsantry bot removed the Waiting for: Product Owner label Apr 25, 2024

azaslavsky added the Waiting for: Community label Apr 25, 2024

getsantry bot added Waiting for: Product Owner and removed Waiting for: Community labels Apr 26, 2024

azaslavsky transferred this issue from getsentry/self-hosted Apr 26, 2024

getsantry bot removed the Waiting for: Product Owner label Apr 26, 2024

getsantry bot added the Waiting for: Product Owner label Apr 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snuba consumes high cpu #5825

Snuba consumes high cpu #5825

samir34ml commented Mar 20, 2024

hubertdeng123 commented Mar 21, 2024

samir34ml commented Mar 24, 2024

csvan commented Mar 25, 2024

chris-laack commented Mar 26, 2024

levani commented Mar 28, 2024 •

edited

Loading

azaslavsky commented Mar 29, 2024

getsantry bot commented Apr 20, 2024

rihards-simanovics commented Apr 24, 2024

azaslavsky commented Apr 25, 2024

rihards-simanovics commented Apr 26, 2024

azaslavsky commented Apr 26, 2024

FrankHeijden commented Apr 30, 2024

Snuba consumes high cpu #5825

Snuba consumes high cpu #5825

Comments

samir34ml commented Mar 20, 2024

Self-Hosted Version

CPU Architecture

Docker Version

Docker Compose Version

Steps to Reproduce

Expected Result

Actual Result

Event ID

hubertdeng123 commented Mar 21, 2024

samir34ml commented Mar 24, 2024

csvan commented Mar 25, 2024

chris-laack commented Mar 26, 2024

levani commented Mar 28, 2024 • edited Loading

azaslavsky commented Mar 29, 2024

getsantry bot commented Apr 20, 2024

rihards-simanovics commented Apr 24, 2024

azaslavsky commented Apr 25, 2024

rihards-simanovics commented Apr 26, 2024

azaslavsky commented Apr 26, 2024

FrankHeijden commented Apr 30, 2024

levani commented Mar 28, 2024 •

edited

Loading