Load Shedding

Load shedding is the deliberate practice of dropping excess requests when a system approaches its capacity limits, ensuring that accepted requests are served with acceptable latency rather than all requests degrading. Unlike rate limiting which enforces policy, load shedding is a self-preservation mechanism triggered by actual resource saturation. By rejecting work early — before it consumes CPU, memory, or threads — the system protects itself from cascading failure and maintains quality of service for the traffic it does handle.

Why This Matters

Every system has a capacity ceiling. When load exceeds that ceiling, latency increases nonlinearly — a server at 95% CPU might have 10x the latency of one at 70%. Without load shedding, all users experience degradation: pages load slowly, API calls timeout, and downstream dependencies pile up queued requests. Load shedding accepts that serving 80% of traffic well is better than serving 100% poorly. By rejecting excess requests with fast 503 responses, the system preserves resources for accepted work. This is especially important during flash crowds, DDoS attacks, or cascading failures when autoscaling cannot react quickly enough. Google's SRE handbook calls load shedding essential for any service operating near its capacity limit.

The Building Blocks

• Admission Control: The decision layer that evaluates incoming requests against current system load and decides whether to accept or reject each one
• Load Signals: Real-time metrics like CPU utilization, queue depth, in-flight request count, or latency percentiles used to detect overload conditions
• Priority Classification: Categorizing requests by importance so that load shedding drops low-priority traffic first while preserving critical operations
• Fast Rejection: Returning 503 Service Unavailable or 429 responses immediately without performing any downstream work to minimize resource consumption
• Feedback Loop: Continuously adjusting the shedding threshold based on observed system behavior to avoid over-shedding or under-shedding

Under the Hood

Load shedding operates as an early-reject mechanism at the ingress point of a service. The simplest implementation tracks in-flight requests using an atomic counter. When the counter exceeds a configured threshold (derived from load testing), new requests are immediately rejected with a 503 status. More sophisticated systems use Little's Law — if average latency rises above a threshold while concurrency is high, the system is overloaded.

Google's Doorman and CoDel-inspired approaches use adaptive algorithms. CoDel (Controlled Delay) monitors the time requests spend waiting in queues. If queue sojourn time exceeds a target (say 5ms) for a sustained interval, the system begins dropping requests. This approach self-tunes: during brief bursts, the queue drains normally, but sustained overload triggers progressive shedding.

Priority-based shedding is essential for production systems. Not all requests are equal — a payment confirmation matters more than a recommendation refresh. Services classify requests into priority tiers (critical, normal, best-effort) and shed lowest priority first. Only if overload persists do higher-priority tiers get affected. Load shedding works best when combined with client-side retry with backoff, ensuring shed requests are retried after a delay rather than immediately re-queuing and worsening congestion.

How Companies Actually Do This

Google Uses CPU-based admission control in its frontend servers, progressively shedding lower-priority requests as CPU utilization exceeds thresholds, documented extensively in their SRE handbook

Netflix Implements priority-based load shedding in Zuul API gateway, dropping non-critical API calls during traffic surges to preserve streaming playback functionality

Amazon Uses shuffle sharding with load shedding in DynamoDB partitions so that hot keys are rejected early without affecting other keys on the same storage node

Common Pitfalls

1. Shedding based on raw request count instead of actual resource utilization — a burst of lightweight requests may be fine while fewer heavy requests cause overload
2. Not differentiating request priority — shedding critical checkout requests alongside analytics pings wastes the opportunity to maintain core business functionality
3. Failing to coordinate shedding across replicas — if load balancers route shed requests to other instances at the same capacity, the entire fleet cascades

Interview Questions Worth Practicing

1. How does load shedding differ from rate limiting and when would you use each?
2. How would you implement priority-based load shedding for a system with mixed critical and non-critical traffic?
3. What signals would you use to detect that a system needs to start shedding load?

The Tradeoffs

• Availability vs. Throughput: Shedding preserves responsiveness for accepted requests but reduces overall throughput by intentionally rejecting work
• Aggressiveness vs. Waste: Shedding too early wastes capacity that could serve requests; shedding too late allows latency degradation before protection kicks in
• Simplicity vs. Fairness: Simple threshold-based shedding is easy but unfair; priority-aware shedding requires request classification infrastructure

How to Explain This in an Interview

Here is how I would explain Load Shedding in a system design interview:

Load shedding is the practice of intentionally rejecting excess requests when a system is near capacity to maintain quality of service for accepted traffic. Unlike rate limiting which enforces business policy, load shedding is a self-preservation mechanism triggered by actual resource saturation. I would implement it using in-flight request counting with Little's Law — when concurrency times latency exceeds capacity, start rejecting. Priority classification is essential: shed analytics requests before payment requests. The system returns fast 503 responses without doing any downstream work, preserving resources. I would pair this with client-side exponential backoff so rejected requests retry gracefully.

Related Topics

• Rate Limiting
• Backpressure
• Bulkhead Pattern
• Autoscaling

Original DosiBridge notes for Load Shedding

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Load Shedding is to connect the concept to a traffic-management problem where overload behavior matters more than the happy-path diagram. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Membership: For Load Shedding, decide the membership before choosing tools; otherwise the design looks clean but fails when load changes.
• Clock Drift: For Load Shedding, decide the clock drift before choosing tools; otherwise the design looks clean but fails when shedding changes.
• Retry Semantics: For Load Shedding, decide the retry semantics before choosing tools; otherwise the design looks clean but fails when load changes.
• Partition Behavior: For Load Shedding, decide the partition behavior before choosing tools; otherwise the design looks clean but fails when shedding changes.
• Reconciliation: For Load Shedding, decide the reconciliation before choosing tools; otherwise the design looks clean but fails when load changes.

Production review question

If a team adopts Load Shedding, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track load-shedding health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Load Shedding, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.