Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding option to force reboot, ignoring active allerts #21

Closed
wants to merge 3 commits into from

Conversation

lunarfs
Copy link

@lunarfs lunarfs commented May 15, 2018

we have a kubernetes cluster running on Azure, sometimes we see high values for softIRQ, we resolve this issue by rebooting the nodes, we want to performe this action even if active prometheus alerts exists. when we see this high values for softIRQ we touches a separate file on the host and the node should restart (despite active alerts exists in prometheus)

Copy link
Contributor

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Seems fine, but I would like @awh to comment on how this fits with the overall concept.

@@ -108,6 +110,18 @@ func sentinelExists() bool {
return false // unreachable; prevents compilation error
}
}
func forceRebootsentinelExists() bool {

This comment was marked as abuse.

This comment was marked as abuse.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@awh What is your take on this , how does this fit the overall roadmap? do you eventually have other suggestions?

@SerialVelocity
Copy link

@lunarfs It might be worth just supporting the word "force" inside the existing sentinel file? It looks like this covers issue #110 I just filed

@evrardjp
Copy link
Collaborator

evrardjp commented Mar 3, 2020

@SerialVelocity that might indeed be simpler, as only the reboot blocked function would need changing by checking the content of the file.

@paprickar
Copy link

I would be also interested to have this merged.
@bboreham / @awh do you think we can have that merged soon?

@github-actions
Copy link

This PR was automatically considered stale due to lack of activity. Please refresh it and/or join our slack channels to highlight it, before it automatically closes (in 7 days).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants