One limiter per server

Your limit is 100 requests a minute and it works perfectly in testing. Production has 4 API servers behind a load balancer. One user sends 400 requests. How many get through?

In 30 seconds

The problem: The limit is 100 a minute and all 400 requests get through.
Why it happens: Each server keeps the count in its own memory and only sees the requests sent to it. Round-robin gives each server 100, so no server ever crosses the limit.
The fix: Keep the count in one shared Redis. The code stays the same; only the place the number lives changes.
In an interview: A limit is only as global as the place its count lives. Keep the counter in one shared store, like Redis, or the limit multiplies by your server count.

Why it matters

It is correct on a laptop, where there is one server. In our run, one server with 150 requests allowed exactly 100.

Behind a load balancer with 4 servers, the same code let 400 of 400 through: 100 per server, each one sure it had enforced the limit. With the counter in shared Redis, exactly 100 of 400 got through. Add servers and the limit grows with them.

Words you need

Rate limit: A cap on how many requests one user can make in a period. Past it, the API answers 429 Too Many Requests.
Load balancer: The component in front of several identical servers that hands each incoming request to one of them, so no single server takes all the traffic.
Atomic: Done as one step that nothing can interrupt. Another request sees it either not started or fully finished.
TTL (time to live): How long a cached copy lives before it expires on its own. It caps how long a wrong copy can survive.

Picture it

The limit is 100 a minute. One user sends 400 requests through a load balancer to 4 servers.

A counter in each server's memory

The load balancer sends 100 requests to each server
Each server counts to 100
No server crosses the limit

400 of 400 get through: the limit, once per server.

One counter in shared Redis

Every server increments the same key
The 101st request sees 101, wherever it lands

100 of 400 get through. The rest get 429s.

The fixes people try

In-memory counter
Fails
new Map() per process
Each server sees only its share of the traffic. The limit multiplies by the number of servers.
Divide the limit by server count
Half a fix
limit / 4 per server
Only works with a fixed server count and a perfectly even load balancer. Autoscaling breaks it.
Sticky sessions
Half a fix
pin each user to one server
Breaks on deploys, scaling and failover, and a client can open new connections.
Shared Redis counter
Works
INCR rate:{user} in one Redis
Every server reads and writes the same number. Where Stripe keeps its buckets.
Limit at the edge
Works
API gateway or proxy
One component sees all traffic before it fans out to servers.

The fix and its cost

Move the count out of the server. Every API server increments the same key in one Redis, so it does not matter which server the load balancer picks.

The code is the same atomic increment as before. Only the place the number lives is different.

What it costs

Redis is now on every request: one extra round trip, and a Redis outage is now the limiter's problem. That is the next trap: should the limiter block or allow when Redis is down?

Interviewers also listen for: local plus global

At very high volume, each server can take a small slice of the budget locally and sync with Redis in batches. Say it as an optimisation with approximate limits, not as the default.

Check yourself

1Why does it pass every test?Show answerHide
Tests run on one server, where the in-memory count sees every request.
2What happens to the limit when you autoscale from 4 servers to 10?Show answerHide
With an in-memory counter it grows to 10 times the limit. With a shared counter it stays the same.
3What new risk does the fix add?Show answerHide
Every request now depends on Redis being reachable, so decide what the limiter does when it is not.

The code

GitHub

limiter.bug.ts

// Each API server process has its own copy of this Map.
const counts = new Map<string, { n: number; resetAt: number }>();

export function allow(user: string, limit = 100) {
  const now = Date.now();
  const entry = counts.get(user);
  if (!entry || entry.resetAt <= now) {
    counts.set(user, { n: 1, resetAt: now + 60_000 });
    return "200 OK";
  }
  entry.n += 1;
  // Correct for the requests THIS server saw. The load balancer sent the rest elsewhere.
  return entry.n > limit ? "429 Too Many Requests" : "200 OK";
}

Example code. The tested version, with a proof test against real Postgres, lands in the repo.

Star the repo for the runnable version

Each trap in this world ships with the bug, the fix and a proof test that fires concurrent requests at a real database.

Star the repo See the folder

Remember

If the count lives in a server, the limit is per server. Put it where every server can see it.