Load shedding reference guide
experimental|
This technology is considered experimental. In experimental mode, early feedback is requested to mature the idea. There is no guarantee of stability nor long term presence in the platform until the solution matures. Feedback is welcome on our mailing list or as issues in our GitHub issue tracker. For a full list of possible statuses, check our FAQ entry. |
Load shedding is the practice of detecting service overload and rejecting requests. By rejecting requests when overloaded, load shedding keeps the application alive. With priority load shedding, the application even keeps working, albeit in a degraded state: only a fraction of requests is handled, others are rejected early. You should consider using it if your application runs in a dynamic environment with a real risk of getting overloaded and is not fronted by another service that sheds load already.
In Quarkus, the quarkus-load-shedding extension provides a load shedding mechanism.
1. Use the load shedding extension
To use the load shedding extension, you need to add the io.quarkus:quarkus-load-shedding extension to your project:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-load-shedding</artifactId>
</dependency>implementation("io.quarkus:quarkus-load-shedding")When added, load shedding is enabled.
It can be disabled by setting quarkus.load-shedding.enabled to false.
There are 2 variants of load shedding: pure and priority-based.
Pure load shedding rejects all requests when overload is detected.
Priority load shedding only rejects a fraction of requests even if overload is detected, based on application-defined criteria.
By default, priority load shedding is enabled.
It can be disabled by setting quarkus.load-shedding.priority.enabled to false.
| Pure load shedding rejects all requests while the application is overloaded, including requests to the non-application endpoints such as health checks. An orchestrator that probes those endpoints may conclude that the application is dead and restart it. Request priority, described below, is the only mechanism that can exempt them. If you disable priority load shedding, consider also exposing the management endpoints on the separate management interface, which is not subject to load shedding. |
There are 2 ways of customizing priority load shedding:
-
By implementing
io.quarkus.load.shedding.RequestPrioritizer, the application can decide which requests can be rejected early on and which requests can only be rejected when there’s no other choice. -
By implementing
io.quarkus.load.shedding.RequestClassifier, the application can classify requests into cohorts which are rejected independently.
Other configuration options are described below, although they should typically be left untouched.
2. The load shedding algorithm
The load shedding algorithm has 2 parts:
-
overload detection
-
prioritization (optional)
2.1. Overload detection
To detect whether the current service is overloaded, an adaptation of the Vegas algorithm (known from TCP) is used.
This algorithm starts with a configurable limit of in-flight requests, by default 100. If the current number of concurrent in-flight requests reaches the limit, overload situation is signalled.
That is not very interesting, but the algorithm dynamically adjusts the limit based on the size of the request queue. If the queue is short, more requests can be handled and the limit increases; if the queue is long, fewer requests can be handled and the limit decreases. The limit cannot increase or decrease indefinitely; there’s a configurable minimum, by default 20, and configurable maximum, by default 1000.
There’s no actual queue of requests that we could monitor, though, so the algorithm estimates the current length of a request queue based on previously seen response times. The longer recent requests take, compared to the recent lowest response time, the longer the queue is supposed to be.
| We know the Vegas algorithm is not the best choice in environments where response times vary. We are working on replacing it with another algorithm, or maybe supporting a small set of algorithms you can select from. |
2.2. Prioritization
If an overload situation is signalled, prioritization is invoked.
If prioritization is disabled, all requests beyond the current limit are rejected immediately. However, by default, prioritization is enabled, which means a request is only rejected if the overload is deep enough.
The overload depth is expressed as a number between 0.0 and 1.0. It is 0.0 when the number of current requests is at the current limit. When the number of current requests exceeds the current limit, the result scales smoothly using the right half of a sigmoid curve (hyperbolic tangent). This provides maximum initial steepness to aggressively shed lower-priority requests immediately upon crossing the limit, while smoothly flattening out near 1.0 to provide a broad runway for higher-priority traffic. When the limit is below the maximum limit, the overload depth reaches 1.0 when the number of current requests reaches the maximum limit. When the limit is at the maximum limit, the overload depth reaches 1.0 when the number of current requests reaches 130% of the maximum limit, leaving a 30% safety buffer for the most important requests.
The more requests are admitted past the limit, the deeper the overload and the more requests are rejected, until the application settles at a level of concurrency it can sustain.
To determine whether a request should be rejected, 2 attributes are considered:
-
request priority
-
request cohort
There are 5 statically defined priorities and 128 cohorts, which amounts to 640 request groups in total.
After both priority and cohort are assigned to a request, a request group number is computed. The group number is smaller for higher priority requests and higher for lower priority requests. If the group number is lower than a threshold derived from the overload depth, the request is accepted, otherwise it is rejected.
The threshold is derived as (1 - overload^3) * 640, where overload is the overload depth (described above) and 640 is the number of request groups (also described above).
The cubic curve is chosen deliberately, because it is flat near 0, so that a slight excess over the concurrency limit only rejects the least important requests.
2.2.1. Customizing request priority
Priority is assigned by a io.quarkus.load.shedding.RequestPrioritizer.
There is 5 statically defined priorities in the io.quarkus.load.shedding.RequestPriority enum: CRITICAL, IMPORTANT, NORMAL, BACKGROUND and DEGRADED.
By default, if no request prioritizer applies, the priority is assumed to be NORMAL.
Further, there’s a special priority level called NEVER, which is exempt from load shedding entirely.
There is one default prioritizer which assigns the priority of NEVER to requests to the non-application endpoints.
It declares no @Priority.
The NEVER priority must be used sparingly.
A request should only be assigned NEVER if it is part of the operational control plane (health checks, in-memory metrics scraping, etc.), where failures would trigger cascading failures, AND if it satisfies strict execution bounds.
The request should execute in sub-millisecond time, only read pre-aggregated values from memory, perform no disk or network I/O, and require no thread offload.
|
It is possible to define custom implementations of the RequestPrioritizer interface.
The implementations must be CDI beans, otherwise they are ignored.
The CDI rules of typesafe resolution must be followed.
That is, if multiple implementations exist with a different @Priority value and some of them are @Alternatives, only the alternatives with the highest priority value are retained.
If no implementation is an alternative, all implementations are retained and are sorted in descending @Priority order (highest priority value comes first).
2.2.2. Customizing request cohort
Cohort is assigned by a io.quarkus.load.shedding.RequestClassifier.
There are 128 statically defined cohorts, with the lowest number being 0 and the highest number being 127.
The classifier should return a number in this interval; if it doesn’t, the result is clamped by a simple bit mask.
This puts requirements on the returned value that are described in the RequestClassifier javadoc.
There is one default classifier which assigns a cohort based on a hash of the remote IP address and current time, such that an IP address changes its cohort roughly every hour.
It declares no @Priority.
If the application is deployed behind a reverse proxy, an ingress or a load balancer, the remote IP address is the address of that intermediary and is therefore identical for all clients.
All requests then land in a single cohort, the cohort dimension collapses and shedding degenerates to all-or-nothing within each priority level.
To avoid this, either set quarkus.http.proxy.proxy-address-forwarding to true, in which case the remote address is taken from the Forwarded or X-Forwarded-For header and the default classifier works as intended, or implement a custom classifier.
A custom classifier can also use a better key than the IP address if the application has one, such as a tenant or account identifier.
|
It is possible to define custom implementations of the RequestClassifier interface.
The implementations must be CDI beans, otherwise they are ignored.
The CDI rules of typesafe resolution must be followed.
That is, if multiple implementations exist with a different @Priority value and some of them are @Alternatives, only the alternatives with the highest priority value are retained.
If no implementation is an alternative, all implementations are retained and are sorted in descending @Priority order (highest priority value comes first).
3. Metrics
If the Micrometer extension is present, this extension publishes the following metrics:
| Name | Type | Description |
|---|---|---|
|
counter |
Number of requests rejected by load shedding. |
|
gauge |
Number of requests currently in flight, as counted by the overload detector. |
|
gauge |
The request limit the overload detector has currently adapted to; reaching it signals an overload. |
|
gauge |
Whether the application is currently overloaded: |
|
gauge |
How deep the overload currently is, as a number between 0.0 and 1.0. |
|
gauge |
How long the current overload has lasted, or |
The overload status is hysteretic, so it cannot be derived from the number of current requests and the current limit; that is why it is published on its own.
Publishing the meters can be turned off by setting quarkus.load-shedding.metrics.enabled to false.
No meters are published if load shedding itself is disabled, because nothing would ever update them.
4. Limitations
The load shedding extension currently only applies to HTTP requests, and is heavily skewed towards request/response network interactions. This means that gRPC, WebSocket and other kinds of streaming over HTTP are not supported. Other "entrypoints" to Quarkus applications, such as messaging, are not supported either.
Further, the load shedding implementation is currently rather basic and not heavily tested in production. Improvements may be necessary.
5. How rejected requests look
An HTTP request is rejected with the 503 Service Unavailable status code and a Retry-After header.
The connection is deliberately left open. Closing it would force the client through a new TCP and TLS handshake, which is expensive and invisible to this load shedding mechanism.
The Retry-After value is derived from how long the overload has already lasted.
The application cannot know when an overload will end, but it does know how long the overload has lasted.
This is used as a proxy: a brief spike is likely over shortly, while an overload that has been going on for minutes is likely to keep going for a while.
A client rejected 2 seconds into an overload is therefore told to come back in a few seconds, while a client rejected 5 minutes in is told to stay away for longer.
The value is capped by quarkus.load-shedding.max-retry-after (120 seconds by default) and is never shorter than 1 second, which is the granularity of the header.
It is also jittered, so that clients rejected in the same instant are not advised to come back in the same instant.
6. Configuration reference
Configuration property fixed at build time - All other configuration properties are overridable at runtime
Configuration property |
Type |
Default |
|---|---|---|
Whether metrics about load shedding should be published. Only takes effect if a metrics extension is present. Environment variable: Show more |
boolean |
|
Whether load shedding should be enabled. Currently, this only applies to incoming HTTP requests. Environment variable: Show more |
boolean |
|
The minimum number of concurrent requests allowed. Must be at least 1 and less than or equal to Environment variable: Show more |
int |
|
The maximum number of concurrent requests allowed. Must be greater than or equal to Environment variable: Show more |
int |
|
The Must be at least 1; the application fails to start otherwise. Environment variable: Show more |
int |
|
The Must be greater than the Environment variable: Show more |
int |
|
The probe factor of the Vegas overload detection algorithm. It controls how often the algorithm discards the lowest request time it has seen and measures against a fresh one; the higher the factor, the rarer that is. Must be positive; the application fails to start otherwise. Environment variable: Show more |
double |
|
The initial limit of concurrent requests allowed. Must be greater than or equal to Environment variable: Show more |
int |
|
The upper bound on the The value actually sent is derived from how long the overload has already lasted, so no lower bound is configurable. This is only a safety measure against telling clients to stay away hours, if the overload has been going on for long. Environment variable: Show more |
|
|
Whether priority load shedding should be enabled. Environment variable: Show more |
boolean |
|
|
About the Duration format
To write duration values, use the standard You can also use a simplified format, starting with a number:
In other cases, the simplified format is translated to the
|
7. Further reading
Netflix Technology Blog:
Uber Engineering Blog:
Amazon Builders' Library:
Google Cloud Blog:
CodeReliant Blog:
TCP Vegas:
-
TCP Congestion Control: A Systems Approach, Chapter 5: Avoidance-Based Algorithms
