Redis goes down, API goes down

Why it matters

Words you need

Rate limit

A cap on how many requests one user can make in a period. Past it, the API answers 429 Too Many Requests.

Fail open

When the limiter itself breaks, let traffic through instead of blocking it, so a limiter outage does not become an API outage.

Cache

A fast copy of data kept close by, here in Redis, so most reads skip the database.

Load balancer

The component in front of several identical servers that hands each incoming request to one of them, so no single server takes all the traffic.

Picture it

Two endpoints behind one limiter: a catalog read that only touches the database, and an OTP send that costs an SMS. Redis goes down.

The fixes people try

• No try/catch

  Fails
  await redis.incr(key)

  Redis down means every request 500s, on every endpoint behind the limiter.

• Fail closed everywhere

  Fails
  catch { return 429 }

  Same outage with a nicer status code. And 429 lies: the user didn't exceed anything.

• Fail open everywhere

  Half a fix
  catch { next() }

  The API stays up, but with no limit at all. A script can burn your SMS budget on the OTP endpoint in a minute.

• Per-endpoint policy

  Works
  catalog → local limit; OTP → 503

  Availability where it's cheap, safety where it's expensive.

• Circuit breaker on top

  Works
  after N failures, skip Redis for 10s

  Stops every request paying the timeout while Redis is down. Goes straight to the fallback.

The fix and its cost

Check yourself

1. 1Why does the catalog fail when only Redis is down?Show answerHide

   The limiter runs before the handler and awaits Redis. Its error becomes a 500 before the catalog code ever runs.

2. 2Why not just fail open everywhere?Show answerHide

   Some endpoints cost money or are abuse targets, like OTP sends. With no limit, one script can do real damage during the outage.

3. 3Why divide the limit by the number of servers in the fallback?Show answerHide

   Each server only sees its own share of traffic. Dividing keeps the total across all servers under the real limit.

The code

import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL!);
const LIMITS = { "GET /catalog": 20, "POST /otp/send": 3 }; // per user per minute

// INCR + EXPIRE in one Lua script (day 5's fix).
const incrWithExpiry = (key: string) =>
  redis.eval('local n = redis.call("INCR", KEYS[1]) if n == 1 then redis.call("EXPIRE", KEYS[1], 60) end return n', 1, key).then(Number);

// Express middleware. Atomic, TTL set: the counter itself is right.
export async function rateLimit(req, res, next) {
  const route = `${req.method} ${req.route.path}`;
  // If Redis is down this throws, the error handler returns 500,
  // and every endpoint behind the limiter goes down with it.
  const count = await incrWithExpiry(`rate:${route}:${req.user.id}`);
  if (count > LIMITS[route]) return res.status(429).send("Too Many Requests");
  next();
}