Infrastructure
Design Rate Limiter
01
What the limiter must do
What the limiter must do
- Limit requests per API key, user, IP address or tenant, with different limits per endpoint.
- Support several limits on the same caller at once, for example 50 per second, 2,000 per minute and 1M per day.
- Allow short bursts above the steady rate, up to a configured bucket size.
- Reject excess requests with HTTP 429 and headers telling the client when to retry.
- Change limits per customer or plan without a deploy, effective within a minute.
- Shadow mode: evaluate a new rule and log what it would block without blocking anything.
- Adds under 2 ms at p99 to each request.
- If the limiter's storage fails, the API keeps serving: fail open with local protection.
- Limits hold across every gateway instance; a client cannot multiply its allowance by spreading requests over hosts.
- Overshoot of up to about 5% under contention is acceptable; large overshoot is not.
- Scales with traffic by adding storage shards and gateway instances, with no single hot node.
Find out whether the goal is fairness between customers, protecting a fragile backend, enforcing a paid quota, or stopping abuse. Each points to a different design. A billing quota must be close to exact and can tolerate a few milliseconds. Backend protection cares about concurrency more than rate. Abuse defence at the edge is coarse and must be extremely cheap. Also ask where the limit is enforced: at the edge, the API gateway, or inside each service.
02
Sizing it
Sizing it
Assume a public API platform that peaks at 1.2 million requests per second, spread across 220 gateway instances in three regions. During any hour, about 40 million distinct callers are active, counting API keys, signed-in users and anonymous IPs. On average each request is checked against two rules.
| Quantity | Working | Result |
|---|---|---|
| Peak request rate | assumption | 1.2M / s |
| Limiter checks at peak | 1.2M x 2 rules, batched into one round trip per request | 2.4M rule evaluations / s, 1.2M calls |
| Per-shard capacity | one Redis primary running a short script | about 120,000 calls / s |
| Shards needed | 1.2M / 120,000, x 2 headroom, across 3 regions | about 24 primaries, each with a replica |
| State per rule per caller | key name about 40 B + one 8 B timestamp + overhead | about 90 B |
| Hot state in memory | 40M callers x 3 rules x 90 B | about 11 GB, well spread |
| Latency budget | same-zone Redis round trip about 0.3 ms, script about 0.05 ms | under 1 ms typical, 2 ms p99 |
| Rejected requests on a normal day | about 1.5% of traffic | about 18,000 / s at peak |
| Decision log volume | sample 1% of allows, keep all rejects | about 30,000 events / s |
Memory and storage are trivial: the entire state fits on one large machine. The design pressure comes from doing a network round trip on every request without adding noticeable latency, and from the fact that the limiter sits in front of everything, so its availability caps the API's availability unless you plan for it failing.
03
Interfaces
Interfaces
The limiter has two interfaces. Internally, gateways call a check function with the caller's identity and the route. Externally, clients see the outcome through status codes and headers. A small admin API manages rules.
A well-behaved client that knows when its allowance resets will wait instead of retrying in a loop. Retry-After turns rejected traffic from a storm into a schedule. Without it, clients guess, usually too early, and each guess is another request you must reject.
04
Where it sits and how it is built
Where it sits and how it is built
flowchart LR
C["Client"] --> EDGE["Edge: coarse per-IP limits"]
EDGE --> GW["API gateway instance"]
GW --> RC["Local rule cache"]
GW --> LF["Local fallback limiter"]
GW -->|"one script call"| RS[("Redis cluster, sharded by key")]
GW -->|allowed| SVC["Backend services"]
GW -->|rejected| R429["429 with headers"]
CFG["Rule config service"] -. "push changes" .-> RC
GW -. "decision events" .-> LOG[["Event stream"]]
LOG --> DASH["Dashboards and alerts"]
SVC --> CL["Concurrency limiter in service"]Enforcement happens in layers, each with a different job. The edge, a CDN or load balancer, applies very coarse per-IP limits and absorbs floods before they reach our network. The API gateway holds the real limiter: it identifies the caller, finds the rules that match the route and plan, and makes one call to a shared Redis cluster to check and update every matching bucket atomically. Inside important services, a separate concurrency limiter caps how many requests run at once, which protects the service when requests suddenly get slow rather than numerous.
Rules live in a config service backed by a database, and every gateway keeps them in memory, refreshed by a push channel with a periodic full reload as a backstop. So looking up which rules apply costs nothing at request time. The only network hop is the Redis call, kept in the same availability zone, and pipelined so that all of one request's rules travel together.
Middleware that builds keys like rl:payments-per-key-second:ak_live_7Hc, calls Redis, and sets headers. It holds a strict 5 ms timeout on that call.
Keys are spread by hash slot, so load spreads evenly across shards. Each check is a short Lua script, which makes read, compute and write a single atomic step.
Stores rules with versions, validates them, and pushes changes to every gateway. Shadow mode lets a new rule run against real traffic before it can block anyone.
An in-memory bucket per key per instance, set to the global limit divided by the instance count. It is used only when Redis is slow or down, so failing open still has a ceiling.
Every reject and a sample of allows go to an event stream. Support teams can see why a customer was limited, and alerts fire if reject rates spike for everyone at once, which usually means a bad rule.
A cap on in-flight requests per service, adapted to observed latency. It catches overload that a rate limit cannot see, such as a slow database making each request hold resources longer.
05
Deep dive: choosing and implementing the algorithm
Deep dive: choosing and implementing the algorithm
Every algorithm answers "how many requests has this caller made recently" in a different way, trading memory per key, accuracy at window edges and how bursts are treated. In an interview, be able to describe all five briefly and defend one.
| Algorithm | State per key | Edge accuracy | Bursts | Where it fits |
|---|---|---|---|---|
| Fixed window counter | One integer | Up to 2x at a boundary | Uncontrolled at edges | Daily quotas where precision at midnight does not matter |
| Sliding window log | One timestamp per request | Exact | Exact | Low limits that must be precise, such as login attempts |
| Sliding window counter | Two integers | Close, assumes even spread | Smoothed | General per-minute limits |
| Leaky bucket queue | A queue of pending requests | Exact output rate | Delayed, not allowed | Smoothing traffic toward a fixed-rate downstream |
| Token bucket as GCRA | One timestamp | Exact for rate and burst | Explicit burst size | Per-second and per-minute API limits |
How GCRA works
A token bucket is usually explained as a bucket that refills at a steady rate and spends one token per request. GCRA stores the same thing differently: instead of a token count plus a last-refill time, it stores one value, the theoretical arrival time (TAT), meaning the moment at which the bucket would be completely full again if no more requests arrived. Each request pushes TAT forward by one emission interval, which is the period divided by the rate. A request is allowed if, after pushing, TAT is no further ahead of now than the burst allowance.
Take a limit of 50 per second with a burst of 10. The emission interval is 20 ms and the burst tolerance is 10 x 20 ms = 200 ms. A caller who has been quiet has TAT in the past, so the first request sets it to now plus 20 ms. Ten rapid requests push it to now plus 200 ms, all allowed. The eleventh would push it to 220 ms ahead, beyond the tolerance, so it is rejected and the caller is told to retry in 20 ms. As time passes, now catches up with TAT and capacity returns smoothly, with no window boundary to exploit.
flowchart TD
REQ["Request reaches gateway"] --> ID["Identify caller: key, user, IP, tenant"]
ID --> RULES["Match rules from local cache"]
RULES --> HOT{"Key flagged as very hot?"}
HOT -->|yes| LOCAL["Spend from locally leased tokens"]
HOT -->|no| LUA["Run GCRA script for all rules"]
LUA --> TO{"Redis answered within 5 ms?"}
TO -->|no| FB["Use local fallback bucket"]
TO -->|yes| DEC{"Every rule allowed?"}
LOCAL --> DEC
FB --> DEC
DEC -->|yes| PASS["Forward with RateLimit headers"]
DEC -->|no| REJ["429 with Retry-After"]Correctness under concurrency
The naive implementation reads a counter, decides in the gateway and writes the new value back. Two gateways doing that at the same instant both read the same value and both allow, which lets a determined client exceed its limit by the number of instances. Running the whole decision as a Lua script on the shard that owns the key removes the race, because Redis executes each script to completion before the next command. Using the shard's clock inside the script also removes clock skew between gateways from the equation.
For a handful of extremely hot keys, such as one huge tenant sending 80,000 requests per second, even a single shard round trip per request is wasteful and puts all that load on one node. For those keys the gateway leases a batch of tokens, say 200 at a time, using the same script with a larger cost, and spends them locally. Overshoot is bounded by the unspent leases across instances, which for this tenant is a fraction of a percent of its limit.
06
Decisions and their costs
Decisions and their costs
The gateway sees every request and the caller's identity, so per-customer limits belong there and only need to be built once. Services add their own in-flight caps because they know their own capacity, which the gateway cannot.
Each layer limits what it understandsTeams end up with different algorithms, different keys and different headers. A client's limits would depend on which service it happens to call, and rejected traffic still reaches every service before it is turned away.
Inconsistent and lateIf Redis is unreachable, the gateway switches to per-instance buckets sized at the global limit divided by instance count. The API stays up, limits stay roughly right, and an outage of the limiter is not an outage of the product.
Availability first, still protectedRejecting everything when the limiter is down turns a cache problem into a full outage. The exception is security-sensitive limits, such as password attempts, where letting traffic through unchecked is worse than refusing it.
Only for login and similarEach region enforces its share of a global limit, sized from that caller's recent traffic mix and rebalanced every 10 seconds. Checks stay in-region at sub-millisecond latency, at the cost of some slack when traffic shifts suddenly.
Fast, approximately globalExactly right, and 70 to 150 ms slower per request for anyone far from the counter's home region. It also makes a single region a dependency for the whole world.
Latency and blast radiusOne 8-byte value per key, exact burst control, no window edges, and a natural retry time to put in headers. It is a little harder to explain than a counter, which is a small price.
Precise and compactTwo counters per key, weighted by how far into the current window you are. Easy to reason about and close enough for most uses, but its accuracy relies on traffic being spread evenly within each window.
A fine second choice07
Failure modes
Failure modes
An early version reads the count with one command and writes with another. Under load, a client firing from many connections gets several times its limit, because instances keep reading the same stale value.
Do the read, the decision and the write in one atomic script on the shard that owns the key, using the shard's own clock.A primary fails and its replica takes several seconds to promote. Every key on that shard times out, and if the gateway waits for the default client timeout, API latency jumps for a twenty-fourth of all callers.
Keep the limiter's timeout at a few milliseconds, then use the local fallback bucket for that key. A circuit breaker per shard stops retries from piling up, and the failover happens quietly behind it.A customer's batch job gets 429s and retries immediately on every thread. Rejected requests now outnumber real ones ten to one, and the limiter itself is the busiest thing in the system.
Always send Retry-After. Make rejections as cheap as possible, with no backend call. Publish client libraries that back off with jitter, and for persistent offenders apply a stricter penalty rule at the edge.With fully independent regional counters, a client that sends traffic to all three regions gets three times its allowance.
Split each global limit into regional budgets based on where the caller's traffic actually arrives, and reconcile every few seconds through a small aggregation job. Strict limits, such as billing quotas, are checked against a single home region.An anonymous per-IP limit blocks a whole university, or a mobile carrier that puts thousands of phones behind one address, while a bot spreading over many IPs is never limited.
Key by the most specific identity available: API key, then user, then device, and IP only as a last resort with generous limits. Raise limits for known shared ranges and use behavioural signals for bots.Someone ships a rule with a missing zero. Within a minute, one of the largest customers sees most requests rejected.
New rules start in shadow mode, changes are versioned with instant rollback, and an alert fires when any single rule's reject rate jumps sharply after a change.08
Mistakes to avoid
Mistakes to avoid
With 220 gateway instances behind a load balancer, a client effectively gets 220 separate allowances, and the limit becomes meaningless.
A sliding log for 1,000 requests per minute per caller means thousands of entries per key across millions of keys.
Clients guess, retry too soon and turn one rejected request into a stream of them.
09
Interview advice
Interview advice
Ask what is being protected. Fair use, backend safety, billing and abuse each lead to different keys, accuracy and failure behaviour. Starting there shows you design for a purpose, not for a buzzword.
Compare the algorithms quickly, then commit. Spend a minute on all five and their trade-offs, then pick one and implement it in detail. Long surveys without a decision score poorly.
Show the race and fix it. Describe two gateways reading the same count, then move the decision into an atomic script. Interviewers almost always probe this.
Decide fail open or fail closed out loud. Explain why most limits fail open with a local ceiling, and why login limits might fail closed.
Handle multiple regions. Mention split budgets and periodic reconciliation, and acknowledge the small overshoot it allows.
Mention concurrency limits. A rate limit does not protect a service whose requests just got ten times slower. A cap on in-flight work does. Few candidates make this distinction.
Finish with the client contract. Show the 429 and the headers you send, and how a well-behaved client uses them.
Sending the remaining allowance on every response, not just on rejections, lets clients slow down before they hit the wall. That cuts reject volume more than any server-side tuning, and it is the detail interviewers say they rarely hear.
10
How the limiter grows
How the limiter grows
A library in the app keeps a token bucket per key in memory. With two instances, limits are off by up to 2x, and nobody minds at this size.
Per-instance limits become meaningless. Move state into one Redis primary with a replica and run the check as a script. Add 429 headers and a simple rules file.
The single Redis primary runs hot. Shard by key, move rules into a config service with shadow mode, add local fallback buckets and token leasing for the hottest keys.
Counters live in each region with budgets rebalanced every few seconds, coarse limits move to the edge, and services add adaptive concurrency limits for overload protection.
Watch and read
Further watching and reading
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.