Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple ProxyURLs in http config #323

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

asuffield
Copy link

This is primarily intended for use with alertmanager, where
notify.Retrier will retry the http request, so if one proxy is not
responding then it will deliver alerts through the other. This avoids a
SPOF at the proxy for alert delivery.

I wish I had a way to write tests for this, but there's no test infra for http proxies and it's remarkably complicated. Sending this for review while I rig up a test-in-production to verify this does what I need.

This is primarily intended for use with alertmanager, where
notify.Retrier will retry the http request, so if one proxy is not
responding then it will deliver alerts through the other. This avoids a
SPOF at the proxy for alert delivery.

Signed-off-by: Andrew Suffield <[email protected]>
@roidelapluie
Copy link
Member

Thank you for this pull request.

I am against this change and this kind of magic. If we should support multiple proxy URL's, I would expect to just pass them as a list to the underlying go stdlib. It is not Prometheus' job to decide or implement that kind of retry mechanisms. This is a huge can of worms and I do not want to open it.

@asuffield
Copy link
Author

I am against this change and this kind of magic. If we should support multiple proxy URL's, I would expect to just pass them as a list to the underlying go stdlib. It is not Prometheus' job to decide or implement that kind of retry mechanisms. This is a huge can of worms and I do not want to open it.

This is just passing them to the underlying go stdlib, which takes a function rather than a list: https://pkg.go.dev/net/http#Transport.

Prometheus does in fact implement this kind of retry mechanism, it's over here: https://github.com/prometheus/alertmanager/blob/main/notify/util.go#L158-L196

@roidelapluie
Copy link
Member

This does not address operability issue. It is hard to know which proxy was used.

as I said, this is a huge can of worms, especially in the alertmanager, when options can be set globally and locally.

To remove the proxy spoc, one would point each alertmanager instance to a different proxy.

@asuffield
Copy link
Author

This does not address operability issue. It is hard to know which proxy was used.

as I said, this is a huge can of worms, especially in the alertmanager, when options can be set globally and locally.

I don't think either of these statements are incorrect, they're just statements about the current alertmanager code, not this PR.

To remove the proxy spoc, one would point each alertmanager instance to a different proxy.

That's assuming we have the same number of proxies and alertmanagers, which could be done by deploying N alertmanager instances per location for N proxies, but is a lot of added complexity compared to just hooking up the standard library feature in the config, like this PR does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants