Add metrics and generic metrics backend

[hubertdeng123]

No description provided.

[aldy505]

10 new containers on this PR. As far as I experienced this, each Snuba container consumes around 50-100 MB of RAM, and each Sentry ingest container consumes around 60-130 MB of RAM. I'd assume this feature would require at least additional 1 GB of RAM tops (assuming every each of it consumes 100 MB of RAM). Should we bump the minimum specs on the requirements?

self-hosted/README.md

Lines 5 to 11 in 8c294f5

	## Requirements
	* Docker 19.03.6+
	* Compose 2.0.1+
	* 4 CPU Cores
	* 8 GB RAM
	* 20 GB Free Disk Space

Edit. Forgot to strike this issue: getsentry/snuba#1670

[BYK]

Or should we finally invest in a unified Snuba instance and unified ingest processes?

[hubertdeng123]

Yep, was pretty painful adding another 10 containers here.

Or should we finally invest in a unified Snuba instance and unified ingest processes?

Let me see what I can do here in the following weeks.

[williamdes]

Or should we finally invest in a unified Snuba instance and unified ingest processes?

That would be cool !
There is already too much containers to look after

And if you want a supervisor alternative, there is https://github.com/FedericoPonzi/Horust that I got into alpine edge testing ;)

[aldy505]

Or should we finally invest in a unified Snuba instance and unified ingest processes?

That would be cool !

There is already too much containers to look after

And if you want a supervisor alternative, there is https://github.com/FedericoPonzi/Horust that I got into alpine edge testing ;)

I don't really think that trimming down the containers to a single container with some kind of supervisor would be different to the consumed resources. As most of us are more likely to have CPU and RAM usage problems instead of storage problems.

Python is a language that requires runtime. If you spawn another app (whether it's on a separate container or same container with supervisor), it will be (most likely) the same, because you'd have to have the runtime consuming some minimum CPU and RAM resource to get it running. I'd argue that it'll be better if we have snuba (and sentry ingest) to have unified consumer/ingester instead of having everything supervised.

[williamdes]

Python is a language that requires runtime.

Yup, but using Horust, well it's Rust ;)
The consumption should be pretty minimal.

I'd argue that it'll be better if we have snuba (and sentry ingest) to have unified consumer/ingester instead of having everything supervised.

It's maybe best not to mix everything, but maybe some tasks could be unified in one container per "topic/section"

[hubertdeng123]

Re: unified snuba/ingest instances

Will have to scope that out, on the radar for Q4:
getsentry/team-ospo#189

[hubertdeng123]

should this live in https://github.com/getsentry/snuba/blob/master/snuba/settings/settings_self_hosted.py instead?

[onewland]

No this is the right place for the setting. This lets sentry know to load sessions metric data from the metrics dataset (which is the change you're trying to make).

Snuba will serve up sessions or metrics and isn't aware of what the product is doing really

[onewland]

This looks right to me, assuming you got it to run locally

[onewland]

No this is the right place for the setting. This lets sentry know to load sessions metric data from the metrics dataset (which is the change you're trying to make).

Snuba will serve up sessions or metrics and isn't aware of what the product is doing really

[hubertdeng123]

Looks good to me locally
Stuff in clickhouse:

Stuff in postgres:

[azaslavsky]

Are these orgs designed to test the new snuba containers somehow?

[hubertdeng123]

those are feature flags that are set at the org level, and utilize the new snuba containers