Health Checks and Failover

Health checks let load balancers, orchestrators and failover systems decide which instances should receive traffic. Failover moves work from a failed component to a healthy one. Getting them right is subtle: checks that are too shallow miss real failures, and checks that are too deep can take down a whole fleet because of one shared dependency.

Kinds of checks

Keep liveness checks trivial (the process can answer HTTP), or a slow dependency will trigger restarts that make things worse. Use readiness to stop traffic to an instance that's warming up, draining for a deploy, or temporarily overloaded.

Typical settings and examples:

• Basic check: a service registry or load balancer calls GET /health every 10 seconds. 200 OK means healthy; a timeout or error counts as a failure, and three failures in a row remove the instance.
• Deep check: the inventory service also tests its database connection and returns 503 Service Unavailable when the database is unreachable, so it is taken out of rotation even though the process is running. Use this with care (see the trap below).
• Passive check: the load balancer watches real responses and marks an instance unhealthy when, say, 10% of its requests fail, catching problems that periodic probes miss.

The shared-dependency trap

If every instance's health check queries the database, a brief database slowdown makes all instances fail their checks at once. The load balancer removes them all, and a partial degradation becomes a total outage. Health checks should mostly reflect the instance itself; failures of shared dependencies are better handled with circuit breakers, fallbacks and alerts.

Many load balancers fail open: if all targets look unhealthy, they send traffic to all of them anyway, on the theory that a widespread failure is more likely a bad health check than every server dying at once. Know how yours behaves.

Detecting failure reliably

• Require several consecutive failures before marking unhealthy (e.g. 3 checks, 5 s apart) and several successes before marking healthy again, to avoid flapping.
• Combine active checks with passive checks (outlier detection on real traffic: eject an instance returning many 5xx or timing out).
• Check from more than one place where possible; a single prober with a network problem can falsely declare everything down.
• In clusters, use gossip-based or consensus-based membership rather than one central checker. See gossip and failure detection.

Failover patterns

Graceful shutdown and draining

Planned failovers (deploys, scale-in) should never drop requests: mark the instance not ready, wait for the load balancer to stop sending new requests, let in-flight requests finish (with a timeout), close long-lived connections politely so clients reconnect elsewhere, then exit.