Distributed Rate Limiting with Redis: Token Buckets Across Pods

Why do per-pod rate limiters break when you scale horizontally?

With per-pod limiting, each replica keeps its own counter in memory. No replica knows what the others have allowed, so the limit you configure is not the limit clients experience.

Illustrative estimate. Assume a target of 400 requests per minute per client, split across N = 4 pods by setting a per-pod limit of L = 100.

• If load spreads evenly, a client can reach N × L = 400. That matches the target only while N stays at 4.
• If the autoscaler grows the deployment to 10 pods, the same client can make 1,000 requests per minute.
• If it shrinks to 2 pods, the client is capped at 200.

The limit now depends on replica count, which changes for reasons unrelated to that client.

Uneven traffic breaks it in the other direction. Long-lived HTTP/2 or keep-alive connections, session affinity, or a skewed hash can pin one client to a single pod. That client hits the 100-request cap while the other three pods hold 300 unused allowance for it, so it gets throttled at a quarter of its intended limit.

Distributed rate limiting with Redis fixes both problems. Every pod checks and decrements one shared bucket per client, keyed by client or tenant ID:

All three pods read and update the same Redis key, so the client's limit stays fixed regardless of how many pods exist or which one receives the request.

You need this if you run more than one replica and must enforce a per-client or per-tenant limit across all of them.

Copy            client-42 requests (any pod may receive them)
                               |
                               v
                  +-------------------------+
                  |      Load balancer      |
                  +-------------------------+
                     |          |         |
           +---------+          |         +---------+
           |                    |                   |
           v                    v                   v
     +-----------+        +-----------+       +-----------+
     |   pod-a   |        |   pod-b   |       |   pod-c   |
     +-----------+        +-----------+       +-----------+
           |                    |                   |
           | check + decrement  | check + decrement | check + decrement
           |                    |                   |
           +---------+          |         +---------+
                     |          |         |
                     v          v         v
          +-------------------------------------------+
          |                   Redis                   |
          |  +-------------------------------------+  |
          |  | ratelimit:client-42  (one bucket)   |  |
          |  +-------------------------------------+  |
          |  | ratelimit:client-77  (one bucket)   |  |
          |  +-------------------------------------+  |
          +-------------------------------------------+

Every pod checks and decrements the same Redis bucket keyed per client, so client-42's limit stays fixed no matter how many pods run or which one receives the request.

Who should use it?

A shared Redis token bucket is worth adopting if most of the following apply:

• You run several replicas of a service behind a load balancer, especially with an autoscaler. Replica count changes over time, and the limit must not change with it.
• You enforce per-client, per-API-key or per-tenant quotas. Public APIs and multi-tenant platforms need one tenant's limit to mean the same thing on every pod, so a noisy tenant cannot take more than its share.
• You protect a shared downstream with a known capacity, such as a database or a third-party API with its own quota. The combined traffic from all pods must stay under that ceiling.
• You already operate Redis, or are willing to. The limiter adds a network call to the request path, and Redis becomes a dependency whose latency and availability you have to monitor.

It is a poor fit for single-replica services, request paths that cannot absorb an extra round trip, and anything that needs exact accounting, such as billing. Later sections cover these cases and the alternatives.

How does it work?

At a high level, every request goes through the same short path:

1. Derive a key. Middleware in each pod builds a bucket key from the dimension being limited, for example the API key or tenant ID, and optionally the route.
2. Make one call to Redis. The middleware calls a preloaded Lua script with EVALSHA, passing the key, the bucket's capacity, its refill rate, and the request's cost.
3. Refill and deduct atomically. Redis runs the script without interleaving other commands. The script reads the stored token count and last-refill timestamp, adds the tokens earned since then using Redis's own clock, and subtracts the cost if enough tokens are available.
4. Return a decision. The script writes the new state, refreshes the key's TTL, and returns whether the request is allowed, how many tokens remain, and how long to wait if it was rejected.
5. Respond. The middleware either passes the request to the handler or returns HTTP 429 with a Retry-After header.
6. Clean up idle state. Keys for idle clients expire on their own once the bucket would be full again, so Redis memory tracks active clients only.

If Redis is slow or unreachable, a timeout and circuit breaker route the request to a predefined failure policy instead: fail open, fail closed, or a local per-pod bucket. The following sections explain each step in detail.

How does a Redis token bucket work?

A token bucket has two parameters. Capacity is the maximum number of tokens the bucket holds, which is also the largest burst a client can send at once. Refill rate is how many tokens are added per second, which sets the sustained throughput. A bucket with capacity 20 and a refill rate of 5 per second allows a burst of 20 requests, then about 5 per second after that.

Nothing refills the bucket in the background. Each key stores only two values: the current token count and the timestamp of the last refill. This is lazy refill. When a request arrives, the limiter computes the elapsed time since that timestamp and multiplies it by the refill rate. It adds the result to the stored count, capped at capacity. It then tries to subtract the request's cost. If enough tokens remain, it writes back the new count and timestamp and allows the request. Otherwise it rejects the request. Idle clients therefore cost nothing between requests.

Why the application can't do the read and write

Suppose two pods each read a bucket holding 1 token. Both see enough to proceed, both deduct, and both write back 0. Two requests got through on one token. Under load, this race repeats and lets traffic exceed the limit.

A Lua script sent with EVAL or EVALSHA closes this gap. Redis runs the whole script without interleaving other clients' commands, so the read, refill, deduct and write happen as one atomic step.

Use Redis's clock

Pod clocks drift apart. A pod whose clock runs ahead sees more elapsed time and creates tokens that should not exist. A pod whose clock runs behind writes an old timestamp, so the next caller sees inflated elapsed time. Calling TIME inside the script gives every pod the same clock.

This is safe under script effects replication, where replicas receive the script's resulting writes rather than re-running the script. Older verbatim replication re-executed scripts on replicas and therefore required them to be deterministic.

How do you implement the atomic token bucket in a Lua script?

The script takes one key, the bucket, and three arguments: capacity, refill rate in tokens per second, and the cost of this request (usually 1). It starts with HMGET on the tokens and ts fields. If both are nil, the bucket is new and starts full.

Next it calls TIME and builds a millisecond timestamp from the seconds and microseconds parts. Reading the clock inside Redis means every pod measures against the same clock, so skew between nodes cannot inflate or drain buckets. Elapsed time is clamped at zero. Refill is min(capacity, tokens + elapsed × rate).

If the refilled balance covers the cost, the script subtracts it and allows the request. Otherwise it leaves the balance unchanged. Either way it writes tokens and ts back with HSET. Then it sets PEXPIRE to roughly capacity / rate plus a margin. After that long an idle bucket would be full anyway, so deleting it gives the same result and keeps idle keys from piling up.

The script returns allowed, remaining and retry-after, where retry-after is (cost − tokens) / rate rounded up. Round the values yourself. Redis truncates Lua floats to integers in replies.

Key design

Name keys after the dimension being limited, for example rl:{tenant:42}:route:orders. Use separate keys for API key, user ID, tenant and route. In Redis Cluster, only the substring inside braces is hashed. Any keys that one script touches together, such as a tenant limit and a per-route limit, must share a hash tag so they land on the same slot. Be aware that a single busy tenant then concentrates its load on one shard.

Keep the script O(1): no loops, no scans, no growing data structures. Redis runs it atomically on its main thread, and every other client waits until it finishes.

Copyimport redis

TOKEN_BUCKET_LUA = """
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])          -- tokens per second
local cost = tonumber(ARGV[3])          -- usually 1

local state = redis.call('HMGET', key, 'tokens', 'ts')
local t = redis.call('TIME')            -- Redis clock: same for every pod
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

-- Missing state means a new (or expired) bucket, which starts full
local tokens = tonumber(state[1]) or capacity
local ts = tonumber(state[2]) or now
local elapsed = math.max(0, now - ts)   -- clamp in case ts is ahead of now
tokens = math.min(capacity, tokens + elapsed * rate / 1000)

local allowed, retry_ms = 0, 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
else
  retry_ms = math.ceil((cost - tokens) * 1000 / rate)
end

redis.call('HSET', key, 'tokens', tokens, 'ts', now)
-- After capacity/rate seconds idle the bucket is full anyway; +1s margin
redis.call('PEXPIRE', key, math.ceil(capacity / rate * 1000) + 1000)

-- Round explicitly: Redis truncates Lua numbers to integers in replies
return {allowed, math.floor(tokens), retry_ms}
"""

r = redis.Redis(host="localhost", port=6379)
token_bucket = r.register_script(TOKEN_BUCKET_LUA)  # EVALSHA with EVAL fallback

allowed, remaining, retry_ms = token_bucket(
    keys=["rl:{tenant:42}:route:orders"], args=[100, 10, 1])

An O(1) Lua token bucket that refills against the Redis server clock, writes state and a TTL, and returns allowed, remaining and retry-after in milliseconds, called from Python with a hash-tagged per-tenant route key.

How do you wire the limiter into service middleware?

The middleware should make exactly one round trip per request. Pass the key, capacity, refill rate, and request cost to the script, and have it return three values: whether the request was allowed, the tokens remaining, and the token deficit when it was rejected. The middleware should not read the bucket separately or do any arithmetic that depends on Redis state, because a second call creates a race window.

On rejection, return HTTP 429 with Retry-After set to ceil(deficit / refill_rate) seconds. Round up, because Retry-After takes whole seconds and rounding down sends clients back before their tokens exist. You can also send informational headers such as X-RateLimit-Limit and X-RateLimit-Remaining on every response. Well-behaved clients use them to pace themselves instead of discovering the limit through 429s.

Weighting expensive endpoints

Cost does not have to be 1. A report export or bulk write might cost 10, so ten cheap reads and one heavy call draw from the same budget. Configure the cost per route, next to the route definition, so reviewers can see it when an endpoint changes.

Cutting per-request overhead

Do not send the script body on every call. Run SCRIPT LOAD at startup, cache the returned SHA, and call EVALSHA. If Redis restarts or fails over and its script cache is empty, it replies with a NOSCRIPT error. Catch that error and retry once with EVAL, which runs the script and repopulates the cache. Use a pooled client so requests don't pay for TCP and auth setup.

The payoff is organizational. One script and one middleware replace each team's hand-rolled limiter. Because the limit lives in Redis rather than in each pod, scaling from 3 replicas to 30 doesn't require anyone to retune per-pod numbers.

Copyimport math
import redis
from functools import wraps
from flask import Flask, request, jsonify, make_response

app = Flask(__name__)
pool = redis.ConnectionPool(host="redis.internal", port=6379, max_connections=50)
r = redis.Redis(connection_pool=pool)

LUA = open("token_bucket.lua").read()  # returns {allowed, remaining, deficit}
SHA = r.script_load(LUA)                # load once at startup, send only the SHA
CAPACITY, REFILL = 100, 10              # bucket size, tokens per second

def take(key, cost):
    try:
        return r.evalsha(SHA, 1, key, CAPACITY, REFILL, cost)
    except redis.exceptions.NoScriptError:
        # Redis restarted or failed over: EVAL runs the script and re-caches it.
        return r.eval(LUA, 1, key, CAPACITY, REFILL, cost)

def rate_limited(cost=1):
    def deco(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            key = "rl:" + request.headers.get("X-API-Key", request.remote_addr)
            allowed, remaining, deficit = take(key, cost)  # the only round trip
            headers = {"X-RateLimit-Limit": str(CAPACITY),
                       "X-RateLimit-Remaining": str(remaining)}
            if not int(allowed):
                # Script returns deficit as a string to keep fractions; round up.
                headers["Retry-After"] = str(math.ceil(float(deficit) / REFILL))
                return jsonify(error="rate limited"), 429, headers
            resp = make_response(fn(*args, **kwargs))
            resp.headers.update(headers)
            return resp
        return wrapper
    return deco

@app.get("/items")
@rate_limited(cost=1)
def list_items(): return jsonify(items=[])

@app.post("/reports/export")
@rate_limited(cost=10)  # heavy endpoint draws 10 tokens from the same budget
def export_report(): return jsonify(status="queued")

Flask middleware that makes one EVALSHA call per request over a pooled Redis client, falls back to EVAL on NOSCRIPT, sets per-route costs, and returns 429 with Retry-After computed as ceil(deficit / refill rate).

How can it save engineering time?

Most of the savings come from removing work that per-pod limiters create over and over:

• No retuning when replica counts change. With per-pod limits, every change to autoscaler bounds or deployment size means recalculating the per-pod numbers, and it is easy to forget. A shared bucket expresses the limit once, per client, and it stays correct at any replica count.
• One implementation instead of many. A single reviewed Lua script and one middleware package replace separate limiters written by each team. Bugs such as non-atomic read-modify-write or clock skew get fixed once rather than rediscovered in every service.
• Limits become configuration. Capacity, refill rate and per-route cost are parameters. Changing a tenant's quota or reweighting an expensive endpoint is a config or code-review change next to the route, not new limiter logic.
• Less time debugging inconsistent throttling. Per-pod limits produce reports like "some requests get 429s and some don't" that depend on which pod a client hit. A shared bucket gives one answer per client, and the bucket's state is a single key you can inspect directly in Redis.
• Fewer support questions from API consumers. Retry-After and the informational rate limit headers tell clients what to do, so they can back off on their own instead of filing tickets.
• Reusable tests and failure policy. The refill unit tests, concurrency test and chaos test described later are written once against the shared script, and the fail-open or fail-closed decision is made once and documented, not reinvented during each incident.

The tradeoff is that you now own a Redis dependency on the request path, which the next sections address.

What should happen when Redis is slow or unavailable?

Every request now depends on a network call, so the limiter needs a decided answer to "what if Redis doesn't respond?" before the incident, not during it.

Fail open lets the request through. This is the usual choice for user-facing APIs, because a short window without limiting beats rejecting all traffic over a problem in a supporting system. It becomes risky when the limiter guards something fragile or expensive, such as a legacy database or a metered third-party API. When Redis disappears, nothing stands between a traffic spike and that dependency.

Fail closed rejects the request. It protects the downstream system, but it turns a Redis incident into a full outage of your service.

Whichever policy you pick, limit how much a failing Redis can cost you:

• Set a tight client timeout, only slightly above the script's normal latency. Otherwise a slow Redis adds latency to every request and can exhaust connection pools.
• Wrap calls in a circuit breaker. After repeated failures, it stops calling Redis for a cooldown period and probes occasionally, so you don't pay the full timeout on every request.
• Optionally fall back to a local in-memory token bucket sized near limit/N, where N is the replica count. Uneven load balancing and autoscaling make it imprecise, but it is a reasonable degraded mode between fully open and fully closed.

Finally, Redis replication is asynchronous. If the primary fails over, the promoted replica may be missing recent bucket writes and will briefly grant extra tokens. That is acceptable for abuse and overload protection. It is also why this limiter should not serve as a billing or quota ledger.

Copyimport time
import redis

CAPACITY, RATE = 100, 50.0   # global bucket: burst size, tokens/sec
REPLICAS = 4                  # N pods sharing the global limit
ON_FAILURE = "local"          # explicit policy: "open" | "closed" | "local"

r = redis.Redis(host="redis.internal", port=6379,
                socket_timeout=0.05, socket_connect_timeout=0.05)
take_token = r.register_script(TOKEN_BUCKET_LUA)  # Lua script from earlier section

class Breaker:
    def __init__(self, threshold=5, cooldown=10.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.open_until = 0, 0.0
    def closed(self):  # after cooldown, one call is let through as a probe
        return time.monotonic() >= self.open_until
    def record(self, ok):
        self.failures = 0 if ok else self.failures + 1
        if self.failures >= self.threshold:
            self.open_until = time.monotonic() + self.cooldown

breaker, local = Breaker(), {}  # local: key -> (tokens, last_refill)

def local_take(key):
    # Degraded mode: each pod enforces roughly limit/N on its own.
    cap, rate, now = CAPACITY / REPLICAS, RATE / REPLICAS, time.monotonic()
    tokens, last = local.get(key, (cap, now))
    tokens = min(cap, tokens + (now - last) * rate)
    ok = tokens >= 1
    local[key] = (tokens - 1 if ok else tokens, now)
    return ok

def allow(key):
    if breaker.closed():
        try:
            result = take_token(keys=[f"rl:{key}"], args=[CAPACITY, RATE])
            breaker.record(True)
            return result == 1
        except redis.exceptions.RedisError:  # includes timeouts and connection errors
            breaker.record(False)
    return {"open": True, "closed": False}.get(ON_FAILURE) if ON_FAILURE != "local" else local_take(key)

A Python limiter wrapper with a tight Redis timeout, a simple circuit breaker, and an explicit failure policy. The policy can fail open, fail closed, or fall back to a per-pod in-memory token bucket sized at limit/N.

What are the alternatives to a token bucket, and when should you avoid this approach?

Token buckets fit most API quotas, but the alternatives below trade burst behavior, memory, and accuracy differently.

A fixed window resets at the boundary. A client can spend its full quota at the end of one window and again at the start of the next. The sliding window counter adds the current count to the previous window's count, weighted by how much of that window still overlaps. That weighting assumes requests were spread evenly, so the result is approximate. The sliding window log stores each request with ZADD, trims expired entries with ZREMRANGEBYSCORE, and counts with ZCARD. It is exact, but memory and trimming work grow with request volume. GCRA tracks a single theoretical arrival time and enforces even spacing. That suits a payment provider or legacy database that fails under bursts.

When to skip a shared limiter

• Single-replica services. An in-process bucket gives the same result without a network hop.
• Tight latency budgets. If the request path cannot absorb an extra Redis round trip, use local limits.
• Protecting local resources. A global quota cannot stop one pod from exhausting its own CPU or memory. Per-pod concurrency limits and load shedding do that job, and they run alongside a shared limiter rather than being replaced by it.

How do you test it, and what does it cost to run at scale?

Testing

Unit-test the refill math. Pass the current time into the limiter as an argument instead of reading the clock inside it. Tests can then step time forward precisely and assert exact token counts: a full bucket after idle time, a partial refill, and no tokens beyond capacity. Wall-clock tests are flaky, and they hide off-by-one errors in the refill calculation.

Run a multi-process concurrency test. Start several processes that hammer one key against a real Redis for a fixed window T. Assert that the total number of allowed requests is at most capacity + rate × T. This test catches non-atomic read-modify-write bugs that single-threaded tests cannot expose.

Run a chaos test. Kill Redis or block its port during load, then assert that the limiter fails open or closed as you designed, and that timeouts stay bounded.

Cost at scale

Every request now waits on a network round trip, and Redis becomes a critical dependency. Redis Cluster maps each key to a single hash slot, so a popular tenant's bucket loads one shard. You can split that bucket into N sub-keys, each with capacity/N, which spreads load across shards but loses accuracy when traffic is uneven. Alternatively, pods can lease tokens in batches. Leasing cuts Redis calls, but enforcement becomes coarser, and tokens held by a pod that dies are lost until the bucket refills.

Illustrative estimate (assumed numbers, not a benchmark): 20 pods × 2,000 req/s × 1 call = 40,000 Redis ops/s. With a lease batch of 50, that drops to 40,000 ÷ 50 = 800 ops/s. Added latency ≈ RTT + script time; with an assumed 0.5 ms + 0.05 ms, that is about 0.55 ms per request.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed rate limiting redis, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.