Set the objective on the tail, measured where the request is assembled
In a fan-out architecture, per-component tail latency compounds into whole-request latency non-linearly, so an objective set on component averages can be met while the assembled request is far slower than any component.
Do this firstAlert on what users feel, not on what machines feel
The measured example is the clearest statement of the problem. In a real fan-out service, a single leaf request’s 99th percentile was 10ms. Waiting for 95% of leaf requests put the 99th percentile at 70ms. Waiting for all of them put it at 140ms — so waiting on the slowest 5% accounted for half the total.
The arithmetic generalises and it is unforgiving. If a single server exceeds one second one time in a hundred, a request fanning out to 100 of them exceeds one second 63% of the time. At one in ten thousand and 2,000 servers, roughly one request in five.
Measure at the point where the results are assembled, because that is the only place the compounding is visible.
The decoy
A mean or median latency dashboard. It is the default panel in every monitoring tool and it is structurally blind to the effect that dominates user-visible latency in any fan-out system.
Evidence
- The Tail at Scale — IIIFan-out amplifies tail latency non-linearly. A single leaf request's 99th percentile was 10ms; for 95% of leaf requests to finish, 70ms; for all of them, 140ms — so the slowest 5% accounted for half the total. With a 1% chance of a single server exceeding one second, a 100-way fan-out exceeds one second 63% of the time.
Last reviewed 2026-08-19.