Chaos Engineering
Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production.
1. The Concept
It is not about "breaking things" randomly. It is about controlled experiments to identify weaknesses before they cause outages.
- Analogy: A vaccine. You inject a small amount of harm (virus) to train the immune system (infrastructure) to handle the real thing.
2. Principles of Chaos Engineering
- Define Steady State: Understand what "normal" behavior looks like (metrics like latency, error rate).
- Hypothesize: "If we disconnect Database Node B, the system should failover to Node C within 2 seconds."
- Introduce Variables: Simulate real-world events (server crash, network latency, disk failure).
- Try to Disprove Hypothesis: If the system fails or steady state is lost, you found a weakness.
3. The Origin: Chaos Monkey
Netflix popularized this practice when moving to AWS. They built Chaos Monkey to randomly terminate instances in production to ensure their engineers built resilient services that could survive server failures automatically.
4. Common Experiments
- Resource Exhaustion: Eat up 100% CPU or RAM on a server.
- Network Latency: Add 500ms delay to all packets between Service A and Service B.
- Packet Loss: Drop 5% of network packets.
- Time Travel: Change the system clock on servers.
- Kill Processes: Terminate critical processes or containers.
5. Tools
- Chaos Monkey: The original random instance terminator.
- Gremlin: A "Chaos-as-a-Service" platform.
- LitmusChaos: Kubernetes-native chaos engineering.
- Chaos Mesh: Open-source cloud-native chaos engineering platform.
6. Safety First
- Blast Radius: Start small (one container, one user group) and expand only if successful.
- Stop Button: Have a big red button to stop the experiment immediately if metrics go haywire.
programming/distributed-systems programming/microservices-architecture programming/software-testing-basics