Designing a Rate Limiter: A Comprehensive Guide

A rate limiter restricts how many requests a client can make in a given window. Without one, a single misbehaving client — malicious or just misconfigured — can take down a service that would otherwise handle load fine.

Why it matters

A few scenarios make the case on their own. A flash sale sends thousands of simultaneous checkout requests and the backend maxes out CPU and memory. An attacker floods an unprotected API, auto-scaling kicks in, and cloud costs spike before anyone notices. A single user hammers an endpoint until database connections are exhausted and every other user's requests slow down. A bot floods a ticket-booking API and grabs every seat before a real user's page even loads.

In each case, a rate limiter — capping requests per user, per IP, or per minute — is what keeps one bad actor from degrading the experience for everyone else.

Three core concepts

Identifier. Every rate limiter needs a key to track requests against — a user ID for authenticated traffic, an IP address for anonymous visitors, a session ID, or a custom key like an organization ID in a multi-tenant system.

Time window. Limits apply over a duration, either a fixed window (100 requests per 60 seconds, resetting on the clock) or a sliding window (counted dynamically over a moving period, which avoids the burst-at-the-boundary problem fixed windows have).

Limit value. The threshold itself, which doesn't have to be a single hard number. A soft limit warns before blocking, a hard limit blocks immediately, and burst capacity allows a temporary spike above the normal rate before enforcement kicks in.

How limits get enforced

• Blocking — requests over the limit are rejected outright, typically with an HTTP 429
• Throttling — requests aren't rejected, just degraded (lower video quality past a data cap, for example)
• Shaping — requests are still processed but at lower priority, common in CDNs that favor paying customers under load

Client-side backoff matters too

Server-side limiting is only half the picture. When a client gets a 429, retrying immediately just makes things worse. Exponential backoff — wait 1 second, then 2, then 4, up to a max — lets the client eventually succeed without adding to the load that triggered the limit in the first place.

Where it lives in the architecture

A common pattern: the load balancer forwards every request to a dedicated rate limiter service before it reaches any backend. That service checks the request against API-key or user-tier limits, queries a fast distributed store like Redis to track counts, and returns either an OK — forwarded on to the real backend — or a 429 that the load balancer returns directly to the client, never touching the backend at all.

Keeping the rate limiter as its own service, backed by a shared cache rather than per-instance memory, is what makes the limits hold up consistently across a fleet of backend nodes instead of resetting per instance.