Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async Healthcheck #301

Merged
merged 19 commits into from
Aug 23, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 5 additions & 1 deletion Gopkg.toml
Original file line number Diff line number Diff line change
Expand Up @@ -108,4 +108,8 @@
# Refer to issue https://github.com/golang/dep/issues/1799
[[override]]
name = "gopkg.in/fsnotify.v1"
source = "https://github.com/fsnotify/fsnotify.git"
source = "https://github.com/fsnotify/fsnotify.git"

[[constraint]]
name = "github.com/InVisionApp/go-health"
version = "v2.1.0"
68 changes: 0 additions & 68 deletions api/healthcheck/composite_indicator.go

This file was deleted.

87 changes: 0 additions & 87 deletions api/healthcheck/composite_indicator_test.go

This file was deleted.

48 changes: 41 additions & 7 deletions api/healthcheck/healthcheck_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,32 +17,36 @@
package healthcheck

import (
h "github.com/InVisionApp/go-health"
"github.com/Peripli/service-manager/pkg/health"
"github.com/Peripli/service-manager/pkg/util"
"net/http"

"github.com/Peripli/service-manager/pkg/health"
"github.com/Peripli/service-manager/pkg/log"
"github.com/Peripli/service-manager/pkg/web"
)

// controller platform controller
type controller struct {
indicator health.Indicator
health h.IHealth
thresholds map[string]int64
}

// NewController returns a new healthcheck controller with the given indicators and aggregation policy
func NewController(indicators []health.Indicator, aggregator health.AggregationPolicy) web.Controller {
// NewController returns a new healthcheck controller with the given health and thresholds
func NewController(health h.IHealth, thresholds map[string]int64) web.Controller {
return &controller{
indicator: newCompositeIndicator(indicators, aggregator),
health: health,
thresholds: thresholds,
}
}

// healthCheck handler for GET /v1/monitor/health
func (c *controller) healthCheck(r *web.Request) (*web.Response, error) {
ctx := r.Context()
logger := log.C(ctx)
logger.Debugf("Performing health check with %s...", c.indicator.Name())
healthResult := c.indicator.Health()
logger.Debugf("Performing health check...")
healthState, _, _ := c.health.State()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignoring error is not good practice

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here second argument is overall status of health, which is not needed since we have custom logic for aggregation, and the error is the potential error for overall health failure (error in some of the indicators). This error is not literally ignored because it is presented in the state of the concrete indicator which failed and is processed in aggregation (included into response).

healthResult := c.aggregate(healthState)
var status int
if healthResult.Status == health.StatusUp {
status = http.StatusOK
Expand All @@ -51,3 +55,33 @@ func (c *controller) healthCheck(r *web.Request) (*web.Response, error) {
}
return util.NewJSONResponse(status, healthResult)
}

func (c *controller) aggregate(overallState map[string]h.State) *health.Health {
if len(overallState) == 0 {
return health.New().WithStatus(health.StatusUp)
}
overallStatus := health.StatusUp
for name, state := range overallState {
if state.Fatal && state.ContiguousFailures >= c.thresholds[name] {
overallStatus = health.StatusDown
break
}
}
details := make(map[string]interface{})
for name, state := range overallState {
state.Status = convertStatus(state.Status)
details[name] = state
}
return health.New().WithStatus(overallStatus).WithDetails(details)
}

func convertStatus(status string) string {
switch status {
case "ok":
return string(health.StatusUp)
case "failed":
return string(health.StatusDown)
default:
return string(health.StatusUnknown)
}
}
Loading