Microservice Rate Limiting with Redis: Per-Tenant Token Buckets

Why in-process limiters fail across replicas, and who needs a shared one

An in-process rate limiter keeps its counters in the memory of one process. That works until the service runs as more than one replica. If each of N replicas enforces "100 requests per minute per tenant" on its own, a tenant whose traffic is spread evenly by the load balancer can make up to N × 100 requests per minute before any replica rejects one. The limit you configured is not the limit you enforce.

Illustrative estimate: with a configured limit of 100 requests per minute and 4 replicas, the effective ceiling is about 400. When the autoscaler scales out to 10 replicas during a traffic spike, the ceiling becomes about 1,000, at exactly the moment you most need to protect downstream dependencies. The quota also depends on load-balancer behavior. Sticky sessions or uneven routing can send one tenant to a single replica, so that tenant is throttled early while others are not.

A distributed rate limiter solves this by moving the counters out of the process and into shared state. Every replica checks and updates the same per-tenant bucket in Redis, so the limit holds no matter how many replicas are running.

This design fits multi-tenant APIs that sell or enforce per-customer quotas, and platform teams running services with many replicas or aggressive autoscaling. You probably don't need it if you run a single instance, if a coarse global limit at the gateway is enough, or if replica counts are fixed and you can divide the limit across them.

Copy BEFORE: in-process limiters (100 req/min per tenant, per replica)

            +---------------------------+
            |       Load balancer       |
            +---------------------------+
            |             |             |
            v             v             v
      +-----------+ +-----------+ +-----------+
      | Replica 1 | | Replica 2 | | Replica N |
      | local ctr | | local ctr | | local ctr |
      | 100/min   | | 100/min   | | 100/min   |
      +-----------+ +-----------+ +-----------+
      tenant A gets up to N x 100/min; N changes as autoscaler acts

 AFTER: distributed limiter (shared per-tenant bucket in Redis)

            +---------------------------+
            |       Load balancer       |
            +---------------------------+
            |             |             |
            v             v             v
      +-----------+ +-----------+ +-----------+
      | Replica 1 | | Replica 2 | | Replica N |
      +-----------+ +-----------+ +-----------+
            |             |             |
            +-------------+-------------+
                          |  check + update
                          v
            +---------------------------+
            |  Redis: tenant:A bucket   |
            | shared by all replicas    |
            +---------------------------+
      tenant A gets 100/min total, for any N

With in-process limiters each replica keeps its own counter, so the effective per-tenant limit is roughly N times the configured value and shifts with autoscaling. With a shared bucket in Redis, every replica enforces the same 100 requests per minute.

How the token bucket works and how it compares to the alternatives

A token bucket has two independent settings. Capacity is the largest burst a tenant can send at once. Refill rate is the sustained throughput the tenant gets over time. A bucket with capacity 100 and refill rate 10/s lets a client send 100 requests immediately, then about 10 per second after that. Keeping the two separate lets you allow short spikes without raising the long-run ceiling.

You don't need a timer to add tokens. Store the token count and the last-update timestamp. On each request, compute tokens = min(capacity, tokens + elapsed × rate), then try to deduct the request's cost. This lazy refill means an idle tenant costs nothing until it sends its next request.

Token buckets suit per-tenant API quotas for three reasons. The state is small and fixed in size, so millions of keys stay cheap. Bursts are explicit and bounded. Weighting expensive endpoints only means deducting more than one token.

Two limits on scope:

• A rate limiter is not a monthly plan quota. Billing-period allowances need durable accounting.
• It is not a fairness mechanism. It caps each tenant but does not divide shared capacity among tenants.

Implementing the bucket as an atomic Redis Lua script

Store each bucket as a hash

Keep one Redis hash per bucket with two fields: tokens (a float) and last_refill (a timestamp in milliseconds). There is no background job that adds tokens. The refill happens lazily whenever a request reads the bucket.

Why GET-then-SET double-spends

Suppose a client reads the bucket, computes the new balance, and writes it back. Two replicas can both read tokens = 1, both decide to allow their request, and both write 0. Two requests got through on one token. With more replicas the overshoot gets worse. A Lua script avoids this because Redis runs each script to completion without interleaving other commands, so the read, check and write happen as one step.

What the script does

The script receives the key in KEYS so it routes correctly in Redis Cluster. It receives capacity, refill rate and request cost in ARGV. On each call it:

1. Calls TIME to get the server's clock. This avoids skew between application hosts.
2. Adds elapsed × rate tokens, capped at capacity.
3. Allows the request if tokens >= cost, then subtracts the cost. A weighted cost lets an expensive endpoint consume several tokens per call.
4. Writes both fields and sets PEXPIRE slightly longer than capacity / rate. An idle bucket would be full by then anyway, so deleting it loses nothing.
5. Returns {allowed, remaining, retry_after_ms}. The retry delay is (cost − tokens) / rate when the request is denied and zero when it is allowed.

Every call does the same fixed amount of work, so the script is O(1).

Calling it from a service

In Python, redis-py's register_script returns a callable that sends EVALSHA first. If the server replies NOSCRIPT, for example after a restart, it falls back to EVAL automatically.

Copyimport redis

TOKEN_BUCKET = """
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2]) / 1000   -- tokens/sec -> tokens/ms
local cost = tonumber(ARGV[3])
local t = redis.call('TIME')           -- server clock, not the app host's
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local b = redis.call('HMGET', KEYS[1], 'tokens', 'last_refill')
local tokens = tonumber(b[1]) or capacity  -- a missing bucket starts full
local last = tonumber(b[2]) or now
-- Lazy refill, capped at capacity; max() guards against a clock step backwards
tokens = math.min(capacity, tokens + math.max(0, now - last) * rate)
local allowed, retry = 0, 0
if tokens >= cost then
  tokens = tokens - cost; allowed = 1
else
  retry = math.ceil((cost - tokens) / rate)
end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'last_refill', now)
-- An idle bucket is full after capacity/rate ms, so expiring it loses nothing
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate) + 1000)
-- Lua numbers become Redis integers, so floor the remaining balance explicitly
return {allowed, math.floor(tokens), retry}
"""

r = redis.Redis(host="redis.internal", port=6379)
# register_script sends EVALSHA and falls back to EVAL on NOSCRIPT
take = r.register_script(TOKEN_BUCKET)

def check(tenant_id: str, cost: int = 1) -> tuple[bool, int, int]:
    # Capacity 100 tokens, refill 50 tokens/sec; key passed via KEYS for Cluster routing
    allowed, remaining, retry_ms = take(
        keys=[f"ratelimit:{tenant_id}"], args=[100, 50, cost]
    )
    return bool(allowed), remaining, retry_ms

ok, remaining, retry_ms = check("tenant-42", cost=5)  # expensive endpoint

A per-tenant token bucket implemented as an atomic Lua script. It uses server time from TIME, refills lazily, applies a weighted cost and sets a TTL, and a redis-py caller invokes it via register_script so EVALSHA is tried first.

Per-tenant quotas, key design and client-facing responses

Quota resolution

Each tenant gets its tier's defaults (capacity and refill rate) unless an override row exists. Overrides belong in the same store as tenant metadata, usually your primary database. Putting them in config files would mean a deploy for every sales exception. Services cache the resolved quota in-process with a short TTL, such as 30–60 seconds. That keeps a database lookup off the hot path while still picking up changes quickly. Pass capacity and rate to the Lua script as ARGV, not hard-coded constants. The script stays generic, and a changed quota takes effect at the next cache refresh with no redeploy.

Key design

The braces are a Redis Cluster hash tag. Only t42 is hashed, so all of a tenant's keys land in the same slot. That lets one script check several buckets atomically. The trade-off is that a very busy tenant concentrates its load on one shard.

Responding to clients

When the bucket is short, return HTTP 429 or gRPC RESOURCE_EXHAUSTED. Set Retry-After to ceil((cost − tokens) / rate) seconds, which is the time until enough tokens have refilled. For gRPC, carry the same value in a RetryInfo error detail. Remaining-quota headers let clients pace themselves before they hit the limit. X-RateLimit-Remaining is a common convention, and an IETF draft defines RateLimit fields.

Clients should add random jitter to Retry-After. Throttled clients all receive the same deadline. Without jitter they retry at the same moment and drain the bucket again in a burst.

Copyimport math, time
import redis

r = redis.Redis(host="localhost", port=6379)
# Generic token-bucket Lua script from earlier; returns {allowed, tokens_left}
bucket = r.register_script(TOKEN_BUCKET_LUA)

TIERS = {"free": (20, 2.0), "pro": (200, 50.0)}  # (capacity, tokens/sec)
CACHE_TTL = 45  # seconds; override changes apply within this window
_cache = {}     # tenant_id -> (expires_at, (capacity, rate))

def resolve_quota(tenant):
    hit = _cache.get(tenant.id)
    if hit and hit[0] > time.monotonic():
        return hit[1]  # hot path: no database lookup
    override = db_fetch_override(tenant.id)  # None or (capacity, rate)
    quota = override or TIERS[tenant.tier]
    _cache[tenant.id] = (time.monotonic() + CACHE_TTL, quota)
    return quota

def check(tenant, cost_class="default", cost=1):
    capacity, rate = resolve_quota(tenant)
    # Hash tag {t42} keeps all of this tenant's buckets in one cluster slot
    key = f"rl:{{{tenant.id}}}:{cost_class}"
    # Limits travel as ARGV, so the script never hard-codes a quota
    allowed, tokens = bucket(keys=[key], args=[capacity, rate, cost, time.time()])
    tokens = float(tokens)
    headers = {
        "X-RateLimit-Limit": str(capacity),
        "X-RateLimit-Remaining": str(max(0, int(tokens))),
    }
    if allowed:
        return 200, headers
    # Seconds until the deficit refills; gRPC would put this in RetryInfo
    headers["Retry-After"] = str(math.ceil((cost - tokens) / rate))
    return 429, headers

Resolving a tenant's quota from a short-lived in-process cache, passing capacity and rate to the Lua script as ARGV on a hash-tagged key, and building a 429 with Retry-After and remaining-quota headers.

Where to enforce: gateway, service middleware, or both

The bucket script is the same wherever it runs. What changes with location is which traffic it sees and how much it knows about the request.

A gateway is a good coarse outer limit. It absorbs abusive bursts before they reach your fleet. It cannot stop a tenant whose single cheap-looking request fans out into dozens of internal calls. It also cannot tell that a report export costs far more than a profile read. Service middleware can do both. The catch is that each team then implements it slightly differently, with different key formats and different header semantics. Package the limiter as a shared library and treat it as a platform dependency, not copied code.

Most teams end up using both: a generous per-tenant ceiling at the gateway and cost-weighted buckets in the services. Two rules apply in either layer:

• Check the bucket before doing expensive work. A rejection issued after you have opened a transaction or called a downstream service saves almost nothing.
• Take the tenant ID from the verified token or session. Never take it from a header such as X-Tenant-Id that the client sets. If clients choose their own key, they choose their own quota.

Failure modes, trade-offs and when to avoid this design

Every rate-limited request now waits on one Redis round trip before it does useful work. Two mitigations reduce that cost, and both give up accuracy:

• Local pre-check. The replica rejects tenants it already knows are empty, without calling Redis.
• Token leases. A replica takes a batch of tokens and spends them locally. A tenant can then exceed its limit by the unspent leases held across all replicas.

When to avoid this design

• Billing-grade accounting. Use a durable ledger, because counters lost in a failover cannot be recovered.
• Long-period plan quotas. Monthly or daily limits need durable storage, since lost Redis state resets them.
• Strict fairness under contention. A token bucket serves requests first come, first served. Use a queue or a weighted fair scheduler instead.
• Very latency-sensitive paths. Prefer an in-process limiter with periodic global synchronization.

Testing, observability, and impact on cost and engineering time

Concurrency test

Run the test against a real Redis instance, not a mock, because the property you are checking is the script's atomicity. Configure one tenant key with a small capacity and a slow refill rate, then fire a few hundred requests in parallel from many goroutines or threads. Record the elapsed time from the first request to the last, and assert:

admitted ≤ capacity + refill_rate × elapsed

If this assertion ever fails, the read-modify-write is no longer atomic. Also assert that admitted is greater than zero, so that a limiter that denies everything does not pass.

What to measure

• Allowed and denied counts per tenant. A sudden rise in denials for one tenant is either abuse or a quota that is set too low.
• Script latency (p50/p99), measured at the client. This is the cost every request pays.
• Redis errors and timeouts, labeled by whether the limiter failed open or failed closed.

Cost and engineering time

Running the limiter adds Redis capacity, one extra network round trip to each request's latency budget, and a new dependency for on-call engineers to understand. In return, every service uses one shared library instead of its own limiter. Limits are read from config, so you can change them without a deploy. A noisy tenant gets throttled before it affects the other tenants.

Illustrative estimate (assumed numbers, not a benchmark). Use these formulas:

Redis ops/s = peak requests/s × enforcement points per request

memory = active tenant keys × bytes per key

Assume a peak of 3,000 req/s, enforcement at both the gateway and the service (2 points), 20,000 active tenants, and about 200 bytes per key:

• Redis ops/s = 3,000 × 2 = 6,000 script calls/s
• memory = 20,000 × 200 B ≈ 4 MB

Under these assumptions, the load is small enough that one modest Redis primary can handle it.

Copy# Concurrency test: run against a real Redis, not a mock, so script atomicity is exercised.
import time, redis
from concurrent.futures import ThreadPoolExecutor

LUA = """
local cap, rate, now = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
-- Callers may arrive slightly out of order; never refill for negative time.
tokens = math.min(cap, tokens + math.max(0, now - ts) * rate)
local ok = 0
if tokens >= 1 then tokens = tokens - 1; ok = 1 end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', math.max(ts, now))
redis.call('EXPIRE', KEYS[1], 60)
return ok
"""

CAPACITY, RATE = 10, 2.0          # small bucket, slow refill (tokens/second)
KEY = "ratelimit:tenant-test-42"  # one tenant key shared by all workers

r = redis.Redis(host="localhost", port=6379)
r.delete(KEY)
take = r.register_script(LUA)

def hit(_):
    return int(take(keys=[KEY], args=[CAPACITY, RATE, time.time()]))

start = time.time()
with ThreadPoolExecutor(max_workers=50) as pool:
    admitted = sum(pool.map(hit, range(500)))
elapsed = time.time() - start

# A non-atomic read-modify-write would let parallel callers overspend the bucket.
assert admitted <= CAPACITY + RATE * elapsed, (admitted, elapsed)
# Guard against a limiter that denies everything and passes trivially.
assert admitted > 0
print(f"admitted={admitted} elapsed={elapsed:.2f}s")

A Python concurrency test that fires 500 parallel requests at one tenant's token bucket in Redis. It asserts the admitted count stays within capacity plus refill over the elapsed time, and that the count is above zero.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice rate limiting redis, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.