the station / sector 05
Observability
How to tell where you areThink of the last incident, and ask how it started: with an alert, or with a human? Then ask a harder one — when the alert fired, could you tell which change caused it without guessing? Knowing something is wrong is the cheap half of this axis.
Whether you can tell what the system is doing, and why.
Observability comes third rather than first because it is how you find out the first two are lying. Fast feedback and cheap rollback are claims about your pipeline; the only evidence that either works in production is instrumentation that watches the real thing.
The common failure is monitoring that answers questions you already knew to ask. Dashboards accumulate around past incidents, which is a form of fighting the last war — the next incident is a question nobody has thought of yet, and answering new questions without shipping new code is the actual capability.
Controls that move you along it
- ›Report accuracy per subgroup, never as one number
- ›Watch the input distribution, and test whether the shift matters
- ›A review with no discussion is not a review
- ›Alert on what users feel, not on what machines feel
- ›Check the data is fit before you consume it
- ›Drive your own deletion and opt-out paths with a canary account
- ›Every deploy leaves a mark in the telemetry
- ›Restore from backup on a normal day, before you have to
- ›Run the suite under sanitizers, not just under the compiler
- ›Set the objective on the tail, measured where the request is assembled
- ›If you inject a fault, hold back a control group