Skip to content
RateLimiting.Adaptive

Rate controllers

A rate controller decides the next rate. The limiter calls it on each throttle and on each success. Two controllers are built in: AIMD default and CUBIC. Select one with the Controller option, or give your own IRateController.

appsettings.json
{
  "AdaptiveRateLimiting": {
    "Limiters": {
      "orders-api": {
        "Controller": "Cubic"
      }
    }
  }
}
The shape of the rate under AIMD and CUBICA sketch. AIMD climbs in equal steps and drops by 30% when it reaches the limit, which gives a saw tooth. CUBIC climbs fast, slows down near the rate of the last cut, then climbs fast again.hidden limitratetime
AIMDCUBICA sketch of the shape only. The steps and the speed depend on the options.

AIMD

Section titled “AIMD”

AIMD means additive increase, multiplicative decrease.

  • On a throttle, the rate is multiplied by Aimd.DecreaseFactor (0.7), down to the minimum.
  • On a success, if Aimd.IncreaseInterval (10 seconds) has passed since the last change, and requests had to wait in it, the rate goes up by Aimd.IncreaseStep (120 units per minute for the whole budget), up to the maximum.

With the defaults, the rate climbs by at most 720 units per minute in each minute. The climb is slow and predictable, and the rate spends most of its time below the limit.

Units used, 60 s
0
Sum of rates
0
Throttled
0
In queue
0
p95 wait
0 s
Rejected or gave up
0

Try these

  1. Saw tooth. Reset and watch the steps of 60 units per minute for each of the 2 instances, and the 30% cut at the limit.
  2. Bigger steps. Use an IncreaseStep of 600. The rate climbs 5 times faster and finds the limit sooner.
  3. Quota drop. Another client takes half of the quota for 60 seconds.
All numbers here come from a model of the library, not from a real API. Rates are in units per minute.

CUBIC

Section titled “CUBIC”

CUBIC uses the window curve of RFC 8312, with the constants of the AWS SDK adaptive retry mode.

  • On a throttle, W is set to the current rate, or to the measured send rate if it is lower. The new rate is W × Cubic.Beta (0.7), down to the minimum.

  • On a success, if requests had to wait, the rate follows this curve. The curve is in units per second:

    K = ∛(W × (1 − β) / C), rate = C × (t − K)³ + W

    W is the rate at the last cut. t is the seconds since the last cut, without the Retry-After pause after it. C is Cubic.ScaleConstant (0.4 for the whole budget). The rate never goes above two times the measured send rate. The rate never goes down on a success, and it stays between the minimum and the maximum.

    The measured send rate is the units of the responses per half-second period, smoothed with a factor of 0.8, as in the AWS SDK.

CUBIC has no increase interval. Each success after a wait can raise the rate.

How CUBIC behaves with the defaults

Section titled “How CUBIC behaves with the defaults”

The curve is fast for rates of hundreds or thousands of units per minute. These are model numbers from the simulator: 2 instances, a hidden quota of 3,000 units per minute, a demand of 2,000 per instance, seed 42. The share of one instance has a maximum of 2,812.5 units per minute.

  • After a cut, K is about 4 seconds. In the first 5 minutes, one instance got back to its maximum after 9 cuts, 8.5 seconds after the cut in the median case (from 6.3 to 12.6 seconds).
  • Before the first throttle, the curve starts at the time the limiter was created, with W equal to the initial rate. The rate rose above the initial share after 5.3 seconds, and reached the maximum after 8.3 seconds. After that, the initial rate has no effect.
  • After a pause, t counts from the end of the pause. After a pause of 13 seconds, the first success raised the rate from 1,113 to 1,421 units per minute (×1.28).
  • Unlike the AWS SDK, the rate goes up only if requests had to wait, and the measured send rate counts units, not requests.

So with the defaults, CUBIC spends most of its time at MaxUnitsPerMinute, and it finds the real limit by a throttle. Set MaxUnitsPerMinute close to, and below, the real limit when you use CUBIC.

Units used, 60 s
0
Sum of rates
0
Throttled
0
In queue
0
p95 wait
0 s
Rejected or gave up
0

Try these

  1. The first seconds. Reset and watch the rate leave the initial value after about 5 seconds.
  2. Maximum below the quota. Set MaxUnitsPerMinute to 2,700, below the hidden quota of 3,000. The rate stays at the maximum with few 429s.
  3. Quota drop. Another client takes half of the quota for 60 seconds.
All numbers here come from a model of the library, not from a real API. Rates are in units per minute.

Compare the modes

Section titled “Compare the modes”

This view runs all four modes on the same seed for 15 minutes. After 5 minutes, another client takes half of the quota for 60 seconds. Change the instances, the quota or the demand, and run it again.

All numbers here come from a model of the library, not from a real API. Rates are in units per minute.

The columns:

Column Meaning
Throughput Units that the API accepted, per minute, over the run.
Share of quota Throughput divided by the mean quota of the run.
429s Throttled responses.
Rejected Requests that the limiter did not send: the queue was full, or the wait was longer than MaxWait.
Gave up Requests that got a 429 on each of their 4 attempts.
p95 wait The 95th percentile of the wait for units, in seconds.
Recovery Seconds after the quota comes back until the accepted units reach 90% of the minute before the drop.

The same comparison for five scenarios. Model numbers, seed 42:

Quota 3,000, 2 instances of 2,000.

Mode Throughput Share 429s Rejected Gave up p95 wait Recovery
No limiter 2,901 100% 64,931 0 9,003 0.0 s 1 s
Fixed rate 1,516 52% 4 20,919 0 70.4 s 0 s
AIMD 2,625 90% 56 11,278 0 44.0 s 8 s
CUBIC 2,890 100% 531 9,306 0 48.5 s 27 s

To get these numbers without a browser, run npm run simulate in the docs folder.

Choosing a controller

Section titled “Choosing a controller”
If Use
You do not know the limit, and 429s are expensive AIMD, the default. It has the fewest 429s when the maximum is above the real limit.
You know the limit, and you can set MaxUnitsPerMinute just below it CUBIC. It gets to the maximum in seconds and stays there.
The quota is large, and AIMD climbs too slowly AIMD with a larger Aimd.IncreaseStep, or CUBIC with a maximum below the limit.
The rate must not change Set InitialUnitsPerMinute, MinUnitsPerMinute and MaxUnitsPerMinute to the same value. The limiter still pauses on Retry-After.
You need a different rule Write an IRateController.

Write your own controller

Section titled “Write your own controller”

Implement IRateController. The controller is pure: it gets a RateControllerState and returns a rate. The limiter does the rest: the lock, the clock, the minimum and maximum, the instance share, the cooldown, the pause, and the rule that a request sent before the last cut does not cut again.

StepController.cs
public sealed class StepController(double step) : IRateController
{
    private static readonly TimeSpan s_interval = TimeSpan.FromSeconds(30);


    public double OnThrottled(in RateControllerState state) =>
        Math.Max(state.Rate / 2, state.MinRate);


    public double? OnSuccess(in RateControllerState state)
    {
        if (!state.Waited || state.SinceLastIncrease < s_interval)
        {
            return null;
        }


        return state.Rate + step / state.InstanceCount;
    }
}

Register it for a named limiter, or pass it to the constructor:

services.AddAdaptiveRateLimiter("billing-api").WithController(new StepController(step: 30));

A custom controller replaces the Controller option. WithController<T>() creates the controller from the container.

The contract

Section titled “The contract”

OnThrottled returns the new rate. The limiter uses the lower of that rate and the current rate, and keeps it at the minimum or above. It calls OnThrottled only when it cuts: on the first throttle, and later only after DecreaseCooldown and for a request sent after the last cut.

OnSuccess returns the new rate, or null for no change. A non-null value, also the same rate, restarts the increase clock and clears the Waited flag. The limiter does not call OnSuccess during a pause.

All rates in the state are for this process, in units per minute:

Property Meaning
Rate The current rate.
MinRate, MaxRate The share of this process of MinUnitsPerMinute and MaxUnitsPerMinute.
RateBeforeLastDecrease The rate before the last cut. Before the first cut, the initial rate.
SinceLastDecrease The time since the last cut. Before the first cut, the time since the limiter was created.
SinceLastIncrease The time since the last change from OnSuccess, or since the last throttle. After a pause, it counts from the end of the pause.
Waited True if a request had to wait for units since the last change.
InstanceCount The number of processes that share the budget. Divide whole-budget steps by it.