Decide in advance what to drop when you run out of capacity
Classifying requests by criticality and shedding the least important first converts a capacity incident from uniform failure into graded degradation — one reported outage held user-initiated availability above 99.4% while prefetch traffic was shed to 20%.
Do this firstSet the objective on the tail, measured where the request is assembled
Overload is not a failure mode you avoid; it is one you choose the shape of in advance.
The mechanism that makes this work is unglamorous: a bounded queue and an early rejection. Oversized queues do not prevent overload, they convert it into latency — a queue sized at ten times the thread count turns a 100ms request into a 1.1 second one once full, with most of that time spent waiting.
What has to exist beforehand is the classification. Deciding which traffic is sheddable during an incident is not a decision anyone makes well at the time, and a limiter with no priority scheme sheds uniformly, which is the behaviour it was installed to prevent.
The decoy
Autoscaling. It buys capacity on a timescale of minutes against an overload that arrives in seconds, and when the dependency you are overloading is the thing that is down, more instances of you does not help.
Evidence
- Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding — IVA concurrency limiter that categorises requests by criticality held user-initiated request availability above 99.4% during a real outage while shedding prefetch traffic to 20% availability.
What would settle it
Whether graded shedding actually reduces user-visible impact across many incidents, rather than in the one that got written up. The available account is a single outage reported secondhand, with no measure of how often the limiter sheds traffic it should have served.
Last reviewed 2026-08-19.