qualitylab

the station

The Tail at Scale

tier III/2013/Communications of the ACM 56(2)

https://www.barroso.org/publications/TheTailAtScale.pdf

Method

"Table 1 lists measurements from a real Google service that is logically similar to this idealized scenario; root servers distribute a request through intermediate servers to a very large number of leaf servers."

Population

"Individual-leaf-request finishing times for a large fan-out service tree (measured from root node of the tree)."

What it does not show

One measured service; the numbers are not shown to generalise. Provides no before-and-after comparison of the mitigations it proposes applied back to this data.

Jeffrey Dean, Luiz André Barroso

Fan-out amplifies tail latency non-linearly. A single leaf request’s 99th percentile was 10ms; for 95% of leaf requests to finish, 70ms; for all of them, 140ms — so the slowest 5% accounted for half the total. With a 1% chance of a single server exceeding one second, a 100-way fan-out exceeds one second 63% of the time.

Tier III: One real production measurement embedded in a broader engineering argument. Concrete measured percentiles rather than pure assertion, but no comparison group.

Cited by