Every deploy leaves a mark in the telemetry
Annotating dashboards with deploy events collapses the most common incident question — "what changed?" — from an investigation into a glance, at a cost of roughly one afternoon.
Do this firstRolling back is one step, and it is practised
The deploy writes an annotation onto the same graphs the on-call watches. Not a separate audit trail — the same picture, with vertical lines on it.
Nearly every incident begins with someone asking what changed, and nearly every minute spent answering that is spent joining two systems by hand. This is a small control with an unreasonable payoff, and it belongs early precisely because it is the thing that tells you whether your other changes helped.
The decoy
A deploy log in a different system. It contains the same timestamps and requires a human to correlate them by eye, at 3am, under pressure, which is when eyes are worst.
Evidence
- Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure — IIICorrelating fault telemetry against in-flight staged rollouts caught 155 critical failures over eight months at 92.4% precision and 100% recall for data-plane rollouts, with the paper reporting that 99.2% of suspicious rollouts were blocked during canary or pilot stages rather than in production.
- Fully Automated DORA Metrics Measurement for Continuous Improvement — IIIDeriving the same delivery metrics from version control, CI and telemetry rather than from a survey exposes wide variation between individual services that a team-level self-report hides: "team performance has limited representational capabilities for individual microservices performance." Also catalogues concrete weaknesses of survey measurement — subjective responses, coarse Likert scales, recall error, and poor scalability.
- Holistic Configuration Management at Facebook — IIIConfiguration change is a first-class incident cause at scale: 16% of high-impact incidents over three months were configuration-related, split roughly 42% obvious errors, 36% subtle load-related issues, and 22% valid changes that exposed latent code bugs. Reports both a canary catching a log spew before rollout and a canary missing a latency regression because the tested scale was too small.
- Automated Canary Analysis at Netflix with Kayenta — IIIAn automated canary analysis system running at real production scale — roughly 200 judgments a day — gating deployments by statistically comparing time-series metrics between a canary and a baseline cluster rather than by eye.
Last reviewed 2026-08-19.