What is per-user rate limiting in a monolith, and what should the limiting key be?
Per-user rate limiting caps how many requests a single identity can make over time. Inside a monolith, every feature shares the same process pool, database connections, and caches. One client running a tight loop can push up latency for everyone else. A per-user limit enforces fairness, slows scraping and credential abuse, and protects shared resources before they saturate. This guide is for backend and platform engineers who own a monolith serving many users.
Choose the limiting key before you choose an algorithm. The key decides whose traffic gets counted together, and a wrong key makes even a well-built limiter punish the wrong people.
| Key | Good for | Risk |
|---|---|---|
| Authenticated user ID | Fairness between individual users | One user with many sessions shares a single budget |
| API key | Integrations and machine clients | Key sharing hides the real consumer |
| Tenant/org ID | B2B plans and contracted capacity | One noisy user can drain the whole team's quota |
| Client IP | Anonymous and pre-auth traffic | Shared NAT, corporate proxies, and mobile carriers group many people under one address |
Resolve the key after authentication but before any expensive work. That means after the session or token is validated, and before database queries, rendering, or calls to downstream services. A request you reject at that point costs almost nothing.
Pre-auth endpoints such as login, signup, and password reset have no user ID yet. Give them their own limits, keyed on IP or on signals such as the submitted username or a device fingerprint. Keep IP thresholds generous enough that an office or campus behind one NAT address does not get locked out.
Who should use per-user rate limiting?
Per-user limiting is worth building if your monolith has these traits:
- Many users share one deployment. All users draw from the same database pool and worker threads. One heavy client can degrade response times for everyone, and you need a way to contain it.
- You expose an API to integrations or scripts. Machine clients retry aggressively, run in loops, and get deployed with bugs. A per-key budget turns a runaway integration into a stream of 429s for that one client instead of an incident for everyone.
- You sell plans or contracted capacity. If different customers are entitled to different throughput, you need per-user or per-tenant enforcement to make those entitlements real.
- You have login, signup, or password-reset endpoints. These are targets for credential stuffing and enumeration, and they need limits even before a user is identified.
- You have expensive endpoints. Search, exports, and report generation can cost far more per call than ordinary reads.
You may not need it yet if the application is an internal tool with a small, trusted user base. The same applies if something in front of the monolith already enforces per-identity limits that match your needs. Even then, per-operation limits on expensive endpoints often belong inside the application, because only the application knows which operations are costly.
How does per-user rate limiting work, end to end?
Every request goes through the same sequence:
- Authenticate. Validate the session or token so the identity is verified rather than taken from a client-controlled header.
- Resolve the key. Pick the user ID, API key, tenant ID, or IP as described above. Combine it with the operation name when the route has its own budget, for example
user:42:export. - Look up the policy. Map the key to a capacity and refill rate from the plan default or a per-user override.
- Check and update the bucket. Read the key's token count and last-refill time, credit tokens for the elapsed time, and spend one if available. A single process can do this in memory. Multiple replicas do it atomically in a shared store such as Redis.
- Admit or reject. If a token was spent, the request continues to the handler. Otherwise it gets a 429 with a
Retry-Aftervalue computed from the bucket state. - Record the outcome. Increment an allowed, limited, or error counter, labelled by route, so you can see who is being throttled and whether the limiter itself is healthy.
The sections below cover each piece: the algorithm, where the checks sit, single-process and Redis storage, failure handling, and the client-facing response.
How does a token bucket work compared to fixed and sliding windows?
A token bucket has two parameters: capacity (the most tokens it can hold) and refill rate (tokens added per second). Each request spends one token, and a request that finds the bucket empty is rejected. Nothing needs to tick in the background. You store only the token count and the last-refill timestamp. On each request you compute elapsed × rate, add that to the count, cap it at capacity, and then try to spend. This lazy refill makes the per-user cost one small record, and that matters when you track millions of keys.
The capacity is a deliberate burst allowance. A user who has been idle can send capacity requests at once, and after that they are held to the refill rate. That suits interactive clients well. It is wrong for a downstream system that cannot absorb spikes, such as a payment provider or a small database pool. For those, use a small capacity or a concurrency limit.
| Approach | State per key | Burst behavior | Main weakness |
|---|---|---|---|
| Fixed window | One counter | Up to 2× limit across a boundary | Boundary bursts |
| Sliding window log | Timestamp per request | Strict, no burst | Memory grows with limit |
| Sliding window counter | Two counters | Smoothed approximation | Slightly inexact |
| Token bucket | Count + timestamp | Up to capacity, by design | Bursts must be sized |
| Concurrency limit | In-flight count | Caps parallelism, not rate | Needs reliable release |
The fixed window's boundary problem works like this. With a limit of 100 per minute, a client can send 100 requests at 0:59 and another 100 at 1:00, so 200 requests land within two seconds.
import time
class TokenBucket:
"""Per-key token bucket with lazy refill. State is just (tokens, last)."""
def __init__(self, capacity: float, rate: float, now: float | None = None):
self.capacity = capacity # max burst size
self.rate = rate # tokens added per second
self.tokens = capacity # start full: an idle user may burst
self.last = time.monotonic() if now is None else now
def allow(self, now: float | None = None) -> bool:
now = time.monotonic() if now is None else now
# Lazy refill: credit tokens for the elapsed time, capped at capacity.
elapsed = max(0.0, now - self.last)
self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1 # spend one token per request
return True
return False # empty bucket: reject
bucket = TokenBucket(capacity=10, rate=1.0, now=0.0)
burst = [bucket.allow(now=0.0) for _ in range(15)]
print("t=0 allowed:", sum(burst)) # 10
later = [bucket.allow(now=5.0) for _ in range(15)]
print("t=5 allowed:", sum(later)) # 5
A minimal in-memory token bucket that stores only a token count and a timestamp, refills lazily from elapsed time on each request, and shows bursts capped at capacity and then held to the refill rate.
Where should the limiter sit in a monolith's request path?
The limiter belongs in two places. Framework middleware applies a coarse ceiling to every request. A finer check near the handler applies a budget specific to the operation. Middleware is good for blanket protection because it runs before any business logic and costs almost nothing per request. However, it only sees a URL and a principal. It does not know that one route returns a cached profile while another builds a CSV export that runs a large database query.
That difference is why expensive endpoints need their own buckets. If search, export, and report generation draw from the same per-user budget as cheap reads, a user can spend their whole allowance on the costliest work. Separate buckets, keyed by user and operation, let you allow many cheap calls and few expensive ones.
Per-user limits alone do not protect shared capacity. Suppose 2,000 active users each stay within 5 requests per second on search (illustrative estimate; both figures are assumed). Together they can still send 10,000 requests per second to a database that handles a fraction of that. A global or per-endpoint ceiling caps the total load regardless of how it is spread across users.
request
│
▼
[ auth ] ──► [ key resolution ] ──► [ global / per-endpoint limit ]
(user, tenant) │
▼
[ per-route, per-user limit ]
│
▼
[ handler ]
both limit checks read and write ──► [ limiter store ]
Auth runs first so the key is a verified identity rather than a header the client controls.
client request
│
┌────────┼── framework middleware: every request ────┐
│ ▼ │
│ ┌────────────┐ ┌─────────────────────────┐ │
│ │ auth │────►│ key resolution │ │
│ │ verified id│ │ (user_id, tenant_id) │ │
│ └────────────┘ └────────────┬────────────┘ │
│ ▼ │
│ ┌───────────────────────────┐ │ ┌──────────────┐
│ │ global/per-endpoint limit │◄──┼─────►│ │
│ │ exceeded ─► 429 │ │ │ limiter │
│ └─────────────┬─────────────┘ │ │ store │
└──────────────────────────────────┬─────────────────┘ │ token state │
│ │ per key │
┌─ near the handler: per route ────┼─────────────────┐ │ e.g. Redis │
│ ▼ │ │ │
│ ┌───────────────────────────┐ │ │ │
│ │ per-route, per-user limit │◄──┼─────►│ │
│ │ exceeded ─► 429 │ │ └──────────────┘
│ └─────────────┬─────────────┘ │
│ ▼ │
│ ┌───────────────┐ │
│ │ handler │ │
│ └───────────────┘ │
└────────────────────────────────────────────────────┘
Request path in a monolith: auth resolves a verified (user, tenant) key, middleware enforces a global or per-endpoint ceiling, a per-route per-user check runs near the handler, and both checks read and write token state in a shared limiter store.
When is an in-process token bucket enough, and when should you avoid it?
If the monolith runs as a single process, the limiter store can be a concurrent map from key to bucket, held in memory. Each bucket needs only a token count and a last-refill timestamp. Refill happens lazily on each request, so no background timer runs per key. Lock per bucket, or shard the map by key hash, so one busy user does not serialize every request behind a global mutex.
The map's weakness is key cardinality. Authenticated user IDs grow slowly. Keys derived from IP addresses do not: a client rotating through addresses creates a new entry per request, and an unbounded map becomes a memory leak an attacker can drive on purpose. Cap the number of entries and evict idle keys. A bucket that has refilled to capacity holds no information, so you can safely drop any entry untouched for longer than its full-refill time. An LRU bound backs this up for bursts of fresh keys.
The approach breaks once you scale out. With N replicas behind a load balancer, each instance enforces the limit independently. A user allowed 100 requests per minute can make up to 100 × N, and no error tells you this is happening. Restarts and rolling deploys also wipe every bucket, so each deploy gives every client a full burst.
The case for starting here is cost. It adds no new dependencies or infrastructure, and the code is small enough to review in one sitting. That saving holds until the deployment topology changes. Once you run a second replica, the limiter needs a shared store.
import threading, time
from collections import OrderedDict
class LocalRateLimiter:
"""Per-key token bucket held in this process only. Not shared across replicas."""
def __init__(self, rate_per_sec, capacity, max_keys=100_000, shards=16):
self.rate = rate_per_sec
self.capacity = capacity
# After this long untouched, a bucket is full again and equals a fresh one.
self.idle_ttl = capacity / rate_per_sec
self.max_per_shard = max(1, max_keys // shards)
# Sharding by key hash: a hot key only contends with its own shard.
self.shards = [(threading.Lock(), OrderedDict()) for _ in range(shards)]
def allow(self, key):
lock, buckets = self.shards[hash(key) % len(self.shards)]
now = time.monotonic()
with lock:
# Unknown keys start full. Refill is computed lazily, with no timers.
tokens, last = buckets.pop(key, (self.capacity, now))
tokens = min(self.capacity, tokens + (now - last) * self.rate)
allowed = tokens >= 1
if allowed:
tokens -= 1
buckets[key] = (tokens, now) # re-insert at the most-recently-used end
# The oldest entry sits at the front. Drop it if it has been idle
# long enough to be full, or if the shard is over its size cap.
# This bounds memory when keys rotate, for example spoofed or rotating IPs.
while buckets:
_, (_, oldest_seen) = next(iter(buckets.items()))
if len(buckets) > self.max_per_shard or now - oldest_seen > self.idle_ttl:
buckets.popitem(last=False)
else:
break
return allowed
limiter = LocalRateLimiter(rate_per_sec=100 / 60, capacity=100)
A single-process token bucket limiter in Python that refills lazily, locks per shard, evicts buckets once they have been idle past their full-refill time, and caps total entries with an LRU bound.
How do you build a Redis rate limiter with atomic Lua scripts?
Once the monolith runs on more than one replica, bucket state has to live somewhere every replica can see, and Redis is the usual choice.
Bucket state and atomicity
Store each user's bucket in a hash such as ratelimit:{user_id} with two fields: current token count and last refill timestamp. Each request needs to read both fields, compute the refill, decide, and write back. If two replicas do that as separate commands, they can both read the same count and both admit a request. A Lua script run with EVAL, or with EVALSHA after loading it once, runs atomically on the server, so no other command touches the key mid-calculation. The script should also set a PEXPIRE of roughly the time it takes a bucket to refill completely. An idle user's key then disappears, and when it is recreated it starts at the same full state the expired bucket would have reached anyway.
Read the time inside the script with Redis TIME instead of passing it in from the application. If each replica passes its own clock, skew between hosts lets a fast clock mint extra tokens.
Hot keys and cost
A single heavy user maps to one key, so in a cluster all of that user's traffic lands on one shard. Every request also adds a network round trip and makes Redis a dependency on the request path. To cut calls, use a local pre-check that rejects requests that are clearly over the limit. Alternatively, have each replica reserve a small batch of tokens and spend them in memory. Either approach reduces Redis traffic, and in exchange a user can briefly exceed the limit by roughly one batch per replica.
import redis
TOKEN_BUCKET_LUA = """
local capacity = tonumber(ARGV[1]) -- max tokens (burst size)
local rate = tonumber(ARGV[2]) -- tokens added per millisecond
local cost = tonumber(ARGV[3]) -- tokens this request needs
-- Use the Redis server clock so replicas with skewed clocks cannot mint tokens.
local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local state = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity -- missing key = full bucket
local ts = tonumber(state[2]) or now
tokens = math.min(capacity, tokens + math.max(0, now - ts) * rate)
local allowed = 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- Expire after a full refill; a recreated key starts full, matching what the bucket would have reached.
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate))
return allowed
"""
r = redis.Redis(host="localhost", port=6379)
take_token = r.register_script(TOKEN_BUCKET_LUA)
def allow(user_id: str) -> bool:
# 20-token burst, refill 10 tokens/s (0.01 per ms), 1 token per request.
key = f"ratelimit:{{{user_id}}}" # renders as ratelimit:{42}
return take_token(keys=[key], args=[20, 0.01, 1]) == 1
A per-user token bucket stored in a Redis hash and updated atomically by a Lua script. The script reads the time with Redis TIME, refills the bucket, decides whether to admit the request, writes the state back and sets a PEXPIRE so idle keys expire.
Should the limiter fail open or fail closed when Redis is unavailable?
Every Redis call in the limiter can fail through a timeout, a refused connection, or a script error. Decide in advance what a failure means, because during an incident there is no time to make that choice.
Failing open lets the request through. The site stays up, but you lose protection at the moment you are most exposed. Redis outages often coincide with load spikes, and an abusive client gets unlimited access until Redis recovers. Failing closed returns a 429 or 503. That keeps the limit strict, but every user now depends on Redis, so a cache outage becomes a full outage.
Most teams choose something in between:
- Tight client timeouts. Give limiter calls a timeout of a few milliseconds and no retries. If Redis is slow and you wait for it anyway, request threads pile up and you get the outage you were trying to avoid.
- Conservative local fallback. When Redis fails, switch to the in-process token bucket, configured at roughly the global limit divided by the replica count. Some protection is better than none.
- Per-route policy. Set the failure behaviour for each route according to what the route risks.
| Route type | On Redis failure | Reason |
|---|---|---|
| Login, password reset | Fail closed | Credential stuffing costs more than brief lockout |
| Writes, expensive jobs | Local fallback | Bounded damage, service stays up |
| Cached reads | Fail open | Cheap to serve, outage costs more |
Instrument the result of each decision. Emit a counter labelled with allowed, limited, error_open, error_closed, and error_fallback, and alert when the error labels are non-zero. If you don't, a fail-open limiter can stay broken for weeks while every request quietly gets through.
How should you return 429s, support tiered limits, and test the limiter?
Returning 429s clients can act on
When a request is rejected, return HTTP 429 with a Retry-After header computed from the bucket's state rather than a fixed guess. If the bucket holds 0.3 tokens and refills at 2 tokens per second, the next whole token arrives in (1 − 0.3) / 2 = 0.35 seconds. Round up to 1, because Retry-After takes whole seconds. On every response, not just rejections, send the same remaining-quota headers, such as limit, remaining tokens (floored), and seconds until full. Clients can then slow down before they hit the limit instead of finding it by failing.
Tiered limits without touching the limiter
The limiter should accept a capacity and refill rate and know nothing about plans. A separate policy lookup resolves a user ID to those two numbers by checking per-user overrides first, then the plan default from config or the database. Cache the result in process with a short TTL so the limiter never adds a database query to the request path. When sales sells a new tier or support raises one customer's ceiling, you change data, not code, and the limiter's tests stay valid.
Testing
Inject the clock into the limiter. A fake clock lets a unit test drain a bucket, advance time by exactly 500 ms, and assert the token count without sleep calls or flaky timing. Load-test two shapes separately. A burst should confirm that capacity is honored, and a sustained run should confirm the steady-state rate converges to the refill rate. Tag allowed, throttled, and error counters by route so you can see which endpoints actually hit the limit.
How can per-user rate limiting save engineering time?
The main saving comes from incidents that never happen or that are diagnosed faster.
- Noisy-neighbour incidents get contained. Without a limiter, a runaway script shows up as general slowness. Someone has to dig through logs to find the client responsible, then block it by hand. With per-user limits, that client gets 429s and everyone else is unaffected. The
limitedcounter, labelled by route, often points straight at the cause. - One component replaces scattered ad hoc checks. Teams without a shared limiter tend to add one-off guards to individual endpoints: a counter here, a sleep there, a hard-coded block list somewhere else. Each guard has to be written, reviewed, and debugged separately. A single limiter with per-route policies replaces them.
- Plan and override changes become configuration. Because the limiter takes only capacity and refill rate, raising a customer's ceiling or adding a tier is a data change. No code change or deploy is needed.
- Tests are fast and deterministic. An injected clock means limiter behaviour can be verified in unit tests rather than chased through flaky timing-dependent tests or production.
- Failure behaviour is decided once. Writing the fail-open, fail-closed, and fallback policy per route ahead of time removes a decision from every future Redis incident.
To size the benefit for your own team, use your own incident history. Count how many past incidents were caused by a single client overloading shared resources. Then estimate the engineer-hours each one took to diagnose and mitigate. Compare that with the cost of building and running the limiter, including the Redis dependency once you scale past one replica. For example, assume four such incidents a year, each taking two engineers half a day. That comes to roughly four engineer-days a year that a working limiter could largely remove (illustrative estimate; all inputs are assumed, not measured).
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to rate limiting per user, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.