Health Check in Microservices

Health Check Definition

A health check in microservices is a mechanism that ensures each service is functioning correctly and is available. It typically involves periodically checking the status of various components of a service and reporting their health.

Use Cases of Health Check

Service Availability Monitoring:

• Scenario: Ensuring that each microservice is up and running.
• Benefit: Helps in quickly identifying and addressing service outages.

Dependency Checking

• Scenario: Verifying that all dependencies of a service are available and functioning.
• Benefit: Ensures the entire application stack is healthy and operational.

Deployment Validation

• Scenario: Checking the health of services post-deployment to ensure they are functioning as expected.
• Benefit: Detects deployment issues early, preventing faulty services from affecting the system.

Load Balancing

• Scenario: Directing traffic only to healthy instances of a service.
• Benefit: Ensures reliable service delivery by avoiding unhealthy instances.

Auto-scaling

• Scenario: Scaling up or down based on the health and load of the services.
• Benefit: Optimizes resource usage and cost efficiency.

Implementation

Tools

• Spring Boot Actuator, AWS Elastic Load Balancer, Kubernetes liveness and readiness probes

Endpoints:

• Health check endpoints (e.g., /health, /status) that return the health status of the service.

API Design:

• Request: GET /health
• Response:

  {
    "status": "DOWN",
    "details": {
      "sql": {
        "status": "DOWN",
        "data": {
          "error": "pq: database 'demo' does not exist"
        }
      },
      "firestore": {
        "status": "UP"
      },
      "kafka": {
        "status": "UP"
      }
    }
  }

Health Status

• UP: Indicates that the application is functioning normally and all health checks have passed.
• DOWN: Indicates that the application is experiencing issues, and one or more health checks have failed.

Implementation of core-go/health

Core Library

• Purpose: Provides basic health check functionalities
• Features:

  • Define standard health check interfaces.

    • Model Health

      package health
      
      type Health struct {
        Status  string                 `json:"status,omitempty"`
        Data    map[string]interface{} `json:"data,omitempty"`
        Details map[string]Health      `json:"details,omitempty"`
      }

  • Allow custom health checks with this standard interface Checker:

    package health
    
    import "context"
    
    type Checker interface {
      Name() string
      Check(ctx context.Context) (map[string]interface{}, error)
      Build(ctx context.Context, data map[string]interface{}, err error) map[string]interface{}
    }

  • Build the response JSON from many custom health checks by this GO function Check

    • This function can be called by http handler (gin, echo, mux, go-chi)

  • Implement basic checks

    • CPU, memory, disk space: not yet implemented.
    • Cache (Redis, Memcached)
    • Databases: sql, mongo, dynamodb, firestore, elasticsearch, cassandra, hive
    • Message Queue
    • External Service Health

  • Integration with Existing Systems, by supporting these Go libraries: gin, echo, mux, go-chi

External Service Health Check Library

• Purpose: Monitors the availability of external services.
• Features:
  • Check HTTP/HTTPS endpoints for expected responses.
    • http client. The sample is at go-sql-hexagonal-architecture-sample.
  • Measure response time and reliability.

Cache Health Check Library

• Purpose: Verifies the status of cache services.
• Features:
  • Check connectivity to cache servers (Redis, Memcached).
    • Redis: support go-redis/redis and garyburd/redigo. The sample is at go-admin.
      • nodejs library for Redis is at redis-plus
  • Validate cache hit/miss ratio and performance metrics.

Database Health Check Library

• Purpose: Monitors the health of database connections
• Features:
  • Check connectivity and response time for various databases (SQL, NoSQL).
    • sql. The sample is at go-sql-sample.
      • nodejs library for My SQL is at mysql2-core. The sample is at sql-modular-sample.
      • nodejs library for Oracle is at oracle-core.
      • nodejs library for Postgres is at pg-extension.
      • nodejs library for MS SQL is at mssql-core.
    • mongo. The sample is at go-mongo-sample.
      • nodejs library for mongo is at mongodb-extension. The sample is at mongo-modular-sample.
    • dynamodb. The sample is at go-dynamodb-tutorial.
    • firestore. The sample is at go-firestore-sample.
    • elasticsearch. The sample is at go-elasticsearch-sample.
    • cassandra. The sample is at go-cassandra-sample.
    • hive. The sample is at go-hive-sample.
  • Provide detailed status messages and error handling.

Message Queue Health Check Library

• Purpose: Ensures message queues are operational.
• Features:
  • Check connectivity and queue depth for different message brokers.
    • Amazon SQS. The sample is at go-amazon-sqs-sample.
    • Google Pub/Sub. The sample is at go-pubsub-sample.
      • health check for nodejs is at google-pubsub. The sample is at pubsub-sample.
    • Kafka. The sample is at go-kafka-sample.
      • health check for nodejs is at kafka-plus. The sample is at kafka-sample.
    • NATS. The sample is at go-nats-sample.
      • health check for nodejs is at NATS. The sample is at nats-sample
    • Active MQ. The sample is at go-active-mq-sample.
      • health check for nodejs is at activemq. The sample is at activemq-sample
    • RabbitMQ. The sample is at go-rabbit-mq-sample.
      • health check for nodejs is at rabbitmq-ext. The sample is at rabbitmq-sample
    • IBM MQ. The sample is at go-ibm-mq-sample.
      • health check for nodejs is at ibmmq-plus. The sample is at ibmmq-sample.
  • Monitor message lag and processing time (Not yet implemented)

Future Libraries to develop

Cluster Health Check Library

• Purpose: Ensures the health of the microservices cluster.
• Features:
  • Check node status, CPU, and memory usage across the cluster.
  • Integrate with orchestration platforms like Kubernetes for liveness and readiness probes.

Metrics and Monitoring Integration Library

• Purpose: Integrates health checks with monitoring tools.
• Features:
  • Export health check results to monitoring systems (Prometheus, Grafana, ELK stack).
  • Provide detailed dashboards and alerting mechanisms.

Notification and Alerting Library

• Purpose: Sends alerts based on health check results.
• Features:
  • Integrate with notification systems (Slack, PagerDuty, email).
  • Provide configurable thresholds and alerting rules.

Integration with Existing Systems

• Designed to integrate seamlessly with existing Go libraries: Gorilla mux, Go-chi, Echo and Gin.
  • handler, to support Gorilla mux and Go-chi. The sample is at go-sql-sample.
  • echo handler to support Echo. The sample is at go-sql-echo-sample.
  • gin handler to support Gin. The sample is at is at go-sql-gin-sample.
    • for nodejs, we have express-ext to integrate with express. The sample is at is at mongo-modular-sample.

Installation

Please make sure to initialize a Go module before installing core-go/health:

go get -u github.com/core-go/health

Import:

import "github.com/core-go/health"