the station / sector 03
Blast radius
How to tell where you areLook at your last three incidents and ask who was affected. If the answer was "everyone" all three times, you have no containment at all — which is worth knowing, because it means every change you ship is a full-population experiment whether you designed it as one or not.
How much of the system, and how many of the users, one mistake can reach.
Containment is what turns an outage into a data point. The same defect that takes down a whole service is, with a cohort boundary in front of it, a graph that looks wrong for four minutes and a rollback nobody had to wake up for.
Worth naming: blast radius is not the same as reliability. A very reliable system with no containment fails rarely and enormously, which is a worse distribution to live under than one that fails often and narrowly, even at the same total downtime. Most teams optimise the average and then get hurt by the tail.
Controls that move you along it
- ›Check the data is fit before you consume it
- ›Deploy dark; release with a flag
- ›No secret is ever committed, and something checks
- ›Test the guarantee your dependency advertises
- ›Model-check the algorithm you cannot test your way out of
- ›Set the objective on the tail, measured where the request is assembled
- ›Turn it on for a cohort before everyone
- ›Decide in advance what to drop when you run out of capacity
- ›If you inject a fault, hold back a control group
- ›Reject an incompatible schema at registration, not at consumption