Chaos testing
Deliberately breaking part of a running system, within a small safety boundary, to prove it keeps working and recovers.
What it is
Chaos testing deliberately injects a failure, such as killing a process, adding network latency, or making a dependency unavailable, while checking that the system still meets a stated steady condition. Netflix built Chaos Monkey during its move onto AWS and open-sourced it in 2012; it made the practice famous by terminating compute instances at random so teams had to design for their disappearance.
Reach for it after ordinary tests, monitoring, and recovery procedures already work. Start with a small blast radius in a controlled environment, state the hypothesis, watch user-facing signals, and stop automatically if the safety boundary is crossed. Move closer to production only as evidence earns it.
Gotcha: random destruction without a hypothesis is an outage generator, not an experiment. A passed pod-kill test also proves only that one failure mode was tolerated under those conditions. DNS trouble, partial responses, clock skew, and slow dependencies can fail very differently.
Ask AI for it
Create a controlled Kubernetes chaos experiment with Chaos Mesh `PodChaos` and `action: pod-kill`. Target one pod with an exact `labelSelectors` rule, keep `mode: one`, and define the steady-state hypothesis as successful synthetic requests plus an error rate below 1%. Wrap it in a Chaos Mesh Workflow so preflight checks run first, the observation window is five minutes, and a conditional branch aborts the run when latency or errors cross the stated limit. Record the pod replacement time, user-visible errors, and recovery time, then write the rollback and cleanup steps beside the manifest.