What is distributed rate limiting with Redis, and why do internal APIs need it?

A distributed rate limiter keeps one shared count of allowed requests that every replica of a service checks before doing work. With Redis, that count lives in a single key, typically a token bucket, and each replica updates it atomically. The limit then applies to the service as a whole rather than to each process.

Per-instance, in-memory limiters fail once a service scales out. Each replica enforces its own copy of the limit, so the effective limit is the configured value multiplied by the replica count. As an illustrative estimate, assume a limit of 100 requests per second per caller and 4 replicas. The caller can actually reach 400 requests per second. If the autoscaler grows the deployment to 12 replicas during a traffic spike, the ceiling becomes 1,200, at the moment the downstream is least able to absorb it. The limit drifts with infrastructure decisions that have nothing to do with capacity. Uneven load balancing makes it worse, because some replicas reject traffic while others still have headroom.

A shared bucket in Redis removes both problems. The limit stays at 100 regardless of replica count, and throttling decisions reflect total traffic instead of whichever pod a request happened to land on.

This matters most for:

  • Platform teams protecting shared downstreams such as databases, search clusters, or third-party quotas with fixed capacity.
  • Internal services with many callers, where you need per-caller fairness.
  • Multi-tenant internal platforms, where one tenant's batch job shouldn't starve others.

A per-instance limiter is enough when the replica count is fixed and you can divide the limit by it, or when the limit is a loose safety net against runaway loops and precision doesn't matter.

 A) Per-replica in-memory limiters: limit x replicas (4 pods -> 400 rps)
    +----------+  +----------+  +----------+  +----------+
    | r1 100/s |  | r2 100/s |  | r3 100/s |  | r4 100/s |
    +----------+  +----------+  +----------+  +----------+
         |             |             |             |
         +-------------+------+------+-------------+
                              v
                   +---------------------+
                   | downstream: 400 rps |
                   +---------------------+

 B) Shared token bucket in Redis: one limit for the whole service
    +----------+  +----------+  +----------+  +----------+
    | replica1 |  | replica2 |  |   ...    |  | replicaN |
    +----------+  +----------+  +----------+  +----------+
         |             |             |             |
         +-------------+------+------+-------------+
                              v
              +-------------------------------+
              | Redis: one bucket key, 100/s  |
              | atomic take-token per request |
              +-------------------------------+
                              |  allowed only
                              v
                   +---------------------+
                   | downstream: 100 rps |
                   +---------------------+

With per-replica limiters the downstream sees the configured limit multiplied by the replica count (illustrative: 100 rps x 4 pods = 400 rps). A single shared token bucket in Redis holds total traffic to 100 rps whether there are 4 replicas or N.

How does the token bucket algorithm work compared with fixed and sliding windows?

A token bucket has three parameters. Capacity is the maximum number of tokens the bucket holds, which sets the largest burst. Refill rate is how many tokens are added per second, which sets the sustained throughput. Cost is how many tokens a request consumes. Cost is usually 1, but you can charge more for expensive calls such as bulk exports. A request is admitted if enough tokens remain. Otherwise it is throttled.

Illustrative estimate (assumed inputs: capacity 100, refill 50 tokens/s, cost 1). An idle caller starts with a full bucket. It can send 100 requests at once, and then it is held to 50 per second. Over a 10-second window, the most it can get through is 100 + 50 × 10 = 600 requests. After draining the bucket, the caller needs 2 seconds of silence to refill it.

That opening 100-request burst is the number to size carefully. Capacity must reflect what the downstream can absorb at once, such as connection pool size, worker count, or queue depth. It is not the average rate multiplied by some factor. A bucket sized only from the average can let through a burst that overwhelms the service it is meant to protect.

Algorithm Burst handling Memory per key Accuracy Complexity
Fixed-window counter Up to 2× limit across a window boundary One counter Coarse Lowest
Sliding-window log No boundary bursts One timestamp per request in window Exact Moderate; memory grows with limit
Sliding-window counter Smoothed Two counters Approximate (weighted estimate) Low
Token bucket Explicit, bounded by capacity Token count + last-refill timestamp Exact for its model Low, but needs atomic read-modify-write

How do you implement a token bucket as an atomic Redis Lua script?

Store each bucket as a Redis hash with two fields: tokens and ts, the last refill time in microseconds. No background job tops buckets up. Each call does a lazy refill instead. It reads both fields, computes the time elapsed since ts, adds elapsed × rate tokens capped at capacity, and then decides. An idle bucket costs nothing until a request touches it.

The read-compute-write sequence must run as one Lua script. Redis executes a script atomically, so two gateway replicas can't both read the last token and both spend it. Pass the bucket key through KEYS rather than building it inside the script. Redis Cluster needs this to route the call.

Take the current time from TIME inside the script, not from the caller. Application hosts drift. A fast clock mints tokens early, and a slow one can produce negative elapsed time. With TIME, the Redis primary is the only clock that matters.

If tokens >= cost, subtract the cost and allow the request. Otherwise store the refilled balance unchanged and reject. Return the wait as (cost - tokens) / rate, which is the token deficit divided by the refill rate. Callers use that value for Retry-After. In both cases, set PEXPIRE to about capacity / rate. An idle bucket would be full again by then, so deleting it loses nothing.

Load the script once with SCRIPT LOAD, keep the SHA1, and call EVALSHA. If you get NOSCRIPT after a restart, failover, or SCRIPT FLUSH, retry with EVAL, which re-caches the script.

Keep the script small and constant-time, with no loops or key scans. While Lua runs, Redis serves no other client.

import redis

TOKEN_BUCKET = """
local capacity, rate, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local t = redis.call('TIME')  -- Redis primary is the only clock
local now = tonumber(t[1]) * 1000000 + tonumber(t[2])  -- microseconds
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or capacity  -- new bucket starts full
local ts = tonumber(b[2]) or now
-- Lazy refill, capped at capacity; clamp guards against negative elapsed
tokens = math.min(capacity, tokens + math.max(0, now - ts) * rate / 1000000)
local allowed, wait = 0, 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
else
  wait = (cost - tokens) / rate  -- seconds until the deficit refills
end
redis.call('HSET', KEYS[1], 'tokens', tostring(tokens), 'ts', string.format('%.0f', now))
-- Idle bucket is full again after capacity/rate seconds, so expire it then
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 1000))
return {allowed, tostring(wait)}  -- string avoids Lua-to-Redis float truncation
"""

r = redis.Redis(host="redis.internal.example", port=6379)
sha = r.script_load(TOKEN_BUCKET)  # cache once, call by SHA1

def take(key, capacity=100, rate=10.0, cost=1):
    args = (capacity, rate, cost)
    try:
        res = r.evalsha(sha, 1, key, *args)
    except redis.exceptions.NoScriptError:
        # Script cache lost (restart, failover, SCRIPT FLUSH); EVAL re-caches it
        res = r.eval(TOKEN_BUCKET, 1, key, *args)
    return int(res[0]) == 1, float(res[1])  # (allowed, retry_after_seconds)

allowed, retry_after = take("ratelimit:{billing-api}:client-42")

A constant-time token bucket Lua script that refills lazily using Redis TIME, consumes or rejects with a computed wait, sets a capacity/rate TTL, and is invoked from Python via EVALSHA with an EVAL fallback on NOSCRIPT.

How should you design rate limit keys for internal callers?

The key decides what a bucket protects. Take caller identity from something the caller cannot forge, such as the SAN in its mTLS certificate or the subject of a verified service token. Never use a self-reported header, because any service could claim another's quota.

Key shape Protects against Weakness
rl:{caller} One noisy service A cheap endpoint and an expensive one share a budget
rl:{endpoint} Overload of one handler One caller can starve the rest
rl:{caller}:endpoint Both More keys and more config

Keep limits in config rather than in code. A lookup such as overrides.get((caller, endpoint)) or overrides.get(caller) or default resolves the most specific entry first. A batch job or a large tenant can then get a different rate without a deploy.

For expensive endpoints, charge more than one token per request. A report export might cost 20 while a point read costs 1. The difficulty is that real cost is often unknown until the work finishes, because it depends on rows scanned or payload size. Charge a conservative estimate up front, and adjust weights later from observed latency.

The braces in the table are Redis Cluster hash tags. The cluster requires every key a script touches to hash to the same slot, and only the text inside {} is hashed. Putting the caller there keeps one caller's keys together for multi-key scripts. It also means a single very busy caller lands on one node. If a key gets that hot, shard its bucket into N sub-keys with N/total capacity each.

import random

# Defaults and overrides come from config, not code. Rate is tokens/sec.
DEFAULT = {"rate": 100, "burst": 200}
OVERRIDES = {
    ("billing-batch", "/reports/export"): {"rate": 5, "burst": 40},
    "search-api": {"rate": 800, "burst": 1600},
}
# Weighted cost: a conservative up-front estimate, tuned later from latency.
COSTS = {"/reports/export": 20, "/items/get": 1}
# Callers hot enough that one Redis node can't absorb them alone.
HOT_SHARDS = {"search-api": 4}

def resolve_limit(caller, endpoint):
    # Most specific entry wins: (caller, endpoint), then caller, then default.
    return OVERRIDES.get((caller, endpoint)) or OVERRIDES.get(caller) or DEFAULT

def bucket_key(caller, endpoint):
    # `caller` must come from the verified mTLS SAN or token subject.
    shards = HOT_SHARDS.get(caller, 1)
    if shards == 1:
        # Only text inside {} is hashed, so all of this caller's keys
        # share one slot and can be used together in a script.
        return f"rl:{{{caller}}}:{endpoint}", 1
    # Shard number goes inside the tag so sub-keys land on different slots.
    n = random.randrange(shards)
    return f"rl:{{{caller}:{n}}}:{endpoint}", shards

def plan_request(caller, endpoint):
    limit = resolve_limit(caller, endpoint)
    key, shards = bucket_key(caller, endpoint)
    return {
        "key": key,
        # Each sub-bucket gets 1/N of the configured capacity.
        "rate": limit["rate"] / shards,
        "burst": limit["burst"] / shards,
        "cost": COSTS.get(endpoint, 1),
    }

Builds a Redis Cluster key with the caller as the hash tag, finds the most specific limit override from config, assigns a weighted cost per endpoint, and splits a hot caller's bucket into N sub-keys that each get 1/N of the capacity.

How should callers handle 429 responses and Retry-After?

When the script returns a deficit instead of granting a token, the deficit already tells you how long the caller must wait. Divide the missing tokens by the refill rate and round up to the next whole second, because Retry-After accepts an integer number of seconds (or an HTTP date). Rounding down sends the caller back before a token exists, and that request is rejected too. Return that value with 429 Too Many Requests. If you want to expose bucket state, add headers for the remaining tokens and the capacity. Names like X-RateLimit-Remaining are a common convention rather than a standard, so document whichever names you choose.

Client-side retry

Clients should treat Retry-After as a minimum wait. When the header is absent, they should fall back to capped exponential backoff: wait a random duration between zero and min(cap, base × 2^attempt). The randomness is what matters here. Without jitter, every caller that was rejected in the same instant retries in the same instant, and the bucket drains again at once. Also cap the total number of attempts, so a sustained overload produces errors the caller can see instead of requests queued indefinitely.

Why this has to be standardized

The limiter only controls how many requests are admitted. It cannot control how many requests arrive. Suppose one internal service retries immediately in a tight loop. Every 429 it receives triggers another request, so the limiter spends its time rejecting traffic, Redis load goes up, and well-behaved callers get fewer tokens. A shared client library that applies Retry-After, jitter and retry caps by default fixes this across the whole fleet. Teams that write their own retry loops tend to get it wrong.

import math
import random
import time

import requests


def rate_limited_response(deficit_tokens, refill_per_sec, remaining, capacity):
    # Round UP so the caller cannot come back before a token exists.
    retry_after = max(1, math.ceil(deficit_tokens / refill_per_sec))
    headers = {
        "Retry-After": str(retry_after),
        # Common convention, not a standard: document the names you pick.
        "X-RateLimit-Remaining": str(int(remaining)),
        "X-RateLimit-Limit": str(capacity),
    }
    return 429, headers, b"rate limit exceeded"


def get_with_retry(url, max_attempts=5, base=0.2, cap=10.0):
    for attempt in range(max_attempts):
        resp = requests.get(url, timeout=5)
        if resp.status_code != 429 or attempt == max_attempts - 1:
            break
        # Full jitter: a random wait in [0, min(cap, base * 2^attempt)].
        delay = random.uniform(0, min(cap, base * 2 ** attempt))
        header = resp.headers.get("Retry-After", "")
        if header.isdigit():
            # Retry-After is a minimum. Add a little jitter so callers
            # rejected together do not all return at the same instant.
            delay = max(delay, int(header) + random.uniform(0, 1))
        time.sleep(delay)
    # Attempts are capped, so sustained overload surfaces as an error.
    resp.raise_for_status()
    return resp

A server helper builds a 429 response whose Retry-After is the token deficit divided by the refill rate, rounded up, and a client retries with capped, jittered exponential backoff that treats Retry-After as a minimum wait and limits the number of attempts.

What happens when Redis is slow or down: fail-open or fail-closed?

Every check adds a Redis round trip to the request path, and every API that enforces limits now depends on Redis. Give the limiter client its own short timeout, a few milliseconds rather than the default socket timeout. A limiter that waits a full second for an answer does more damage than one that never answers. When the call times out or errors, you need a policy decided in advance, not left to whatever exception handling happens to do.

Policy On Redis failure Protects downstream? Main risk
Fail-open Allow the request No Overload hits the backend while protection is off
Fail-closed Return 429 or 503 Yes A Redis outage becomes an API outage
Local fallback In-process bucket with limit ÷ replica count Roughly Limits drift as replicas scale or traffic skews

Most internal APIs fail open for low-risk routes and use a local fallback for expensive ones. Fail-closed suits only downstreams that cannot survive a burst at all. Emit a metric whenever the fallback path runs, so you can see when you are running without protection.

Even while Redis is healthy, the limiter is approximate. With asynchronous replication, a failover can promote a replica that missed recent writes, which refills buckets. Treat the limit as overload protection, not a billing-grade quota.

For very high-throughput callers, a round trip on every request is too expensive. Pre-limit locally, or have each instance take tokens in batches, say 50 at a time, and spend them from memory. This trades some precision for far fewer Redis calls.

How do you test a Redis rate limiter, and when should you use something else?

Testing the script

Run unit tests against a real Redis instance, such as a throwaway container. A mock will not reproduce Lua atomicity or key expiry. If the script accepts the current time as an argument, tests can move the clock forward without sleeping. Cover four cases:

  • Refill math: drain the bucket, advance time by half the refill interval, and assert the exact fractional token count.
  • Cost greater than capacity: the request must be rejected immediately and must never succeed, however long the caller waits.
  • TTL expiry: after an idle period, the key should be gone, and the next call should see a full bucket.
  • Concurrency: with refill disabled, fire many parallel requests at a bucket of capacity C. Exactly C should succeed.

Then load-test with many concurrent callers over a fixed window. As an illustrative estimate, a 100-token bucket refilling at 10 per second should admit about 100 + 10 × 60 = 700 requests over 60 seconds. A large deviation usually points to clock skew or a non-atomic code path.

When to use something else

Alternative Use it when
Per-instance limiter Instance count is stable and an approximate global limit is acceptable
Gateway or service mesh limit Limits are coarse and should be enforced before the request reaches your code
Concurrency limit or load shedding Downstream capacity depends on requests in flight, as with database connections or slow calls
Queue-based smoothing Work can be deferred, and bursts should be absorbed instead of rejected

A single shared script and client library gives every service the same semantics and the same 429 behaviour. That costs less engineering time than each team writing its own limiter. The trade-off is that someone has to own Redis: its availability, monitoring and capacity become a dependency of every service that uses the limiter.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed rate limiting redis, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.