Chaos Engineering and Resilience Testing

Chaos engineering is the practice of deliberately injecting failures into a system, carefully and in a controlled way, to find weaknesses before they cause real outages. Instead of hoping that failover, retries and timeouts work, you run experiments that prove they do: kill an instance, add network latency, fail a dependency, and check whether users notice. It turns resilience from an assumption into evidence.

The experiment loop

1. Define steady state in user-facing terms: "checkout success rate stays above 99.5% and p99 latency below 800 ms".
2. Form a hypothesis: "if one of three payment-service instances is terminated, steady state holds because the load balancer routes around it within 10 seconds".
3. Inject a realistic fault with a limited blast radius.
4. Observe steady-state metrics and the system's behaviour.
5. Stop automatically if steady state breaks beyond a threshold.
6. Learn and fix: if the hypothesis failed, you found a weakness cheaply. Fix it and run the experiment again, then automate it.

Faults worth injecting

Controlling the blast radius

• Start in staging, then in production with a tiny scope: one instance, one percent of traffic, one internal user group.
• Run during working hours with the owning team present and incident tooling ready.
• Define abort conditions that stop the experiment automatically when steady-state metrics degrade.
• Announce experiments and keep a log, so they're not confused with real incidents.
• Widen the scope only after the system passes at the smaller one.

Chaos engineering in production without good observability is just causing outages. You need clear steady-state metrics and the ability to stop instantly before you start.

Game days

A game day is a planned exercise where a team simulates a major failure, such as losing a database primary or a whole region, and practises the response: detection, communication, runbooks and recovery. Game days test people and processes as much as software, and often reveal stale runbooks, missing permissions, and alerts that never fire. They're the best way to know whether a disaster recovery plan would actually work.

Growing the practice

• Begin with experiments against known risks from past incidents, since postmortems are a ready list of hypotheses.
• Automate passing experiments to run continuously, so regressions in resilience are caught like failing tests.
• Track findings and fixes; the value is in the weaknesses found and removed, not the number of experiments run.