Alert on what users feel, not on what machines feel
Paging on user-visible symptoms rather than internal causes reduces alert volume and raises the fraction of pages that correspond to a real problem, which is the number that determines whether anyone still reads them.
Do this firstEvery deploy leaves a mark in the telemetry
Page when requests are failing, when they are slow, when the thing a person came to do cannot be done. Graph the causes; do not wake anyone for them.
The mechanism is attention. An alert set that fires for conditions the system handles trains the on-call to close pages without reading them, and that habit does not distinguish between the noisy alerts and the one that mattered. Alert fatigue is not a morale problem, it is a detection failure with a human in it.
The decoy
A comprehensive set of resource alerts. CPU, memory, disk, queue depth — each one a cause that sometimes produces a symptom, firing at 3am for a condition the system absorbed on its own.
Evidence
- How to Fight Production Incidents? An Empirical Study on a Large-scale Cloud Service — IIQuantifies the detection gap in a real large service: "≈17% of incidents either lacked monitors or telemetry coverage" and "10% incidents were not detected due to bugs, e.g., high threshold, buggy feature, wrong configuration". Correlating cause with detection: "70% incident with code bugs does not have monitors".
- Monitoring Distributed Systems (SRE Book, ch. 6) — IVNames the distinction the control rests on — "Your monitoring system should address two questions: what's broken, and why? The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause" — and the fatigue mechanism: "When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a 'real' page that's masked by the noise."
- Meaningful Availability — IIIA success-ratio availability metric is measurably biased toward the most active clients — 1% of users generate 62% of events — and that bias caused it to over- and under-state real outage severity compared with a per-user metric. What you count decides whether your alerting reflects what users experienced.
- Google Cloud and Workspace global outage, 12 June 2025 — IIIThe inverse case: a code path shipped WITHOUT a flag sat globally live-but-dormant for about two weeks, and when an unrelated policy change activated it everywhere at once it crashed Service Control worldwide, taking down 50+ services. Google's own remediation was to make flags mandatory — to "enforce all changes to critical binaries to be feature flag protected and disabled by default".
Seen in the wild
Anonymised field observation. Illustration, not evidence — these carry no tier and cannot raise one.
A health check answered 200 while the service it fronted had come back with an empty configuration and every feature off. Everything the machine could feel was fine. Everything a user could feel was not.
Last reviewed 2026-08-19.