What is per tenant rate limiting, and why do edge or IP limits fail in multi-tenant SaaS?
Per tenant rate limiting caps how much work each customer account can trigger in a given period, no matter how many users, devices or IP addresses that account uses. In a multi-tenant SaaS application, every tenant shares the same database pool, worker threads and cache. Without a per-tenant cap, one customer running a bulk import or a misconfigured integration can use up that shared capacity, and every other tenant sees higher latency. This is the noisy neighbor problem, and the goal is to contain it rather than just detect it.
Edge and IP-based limits fall short for three reasons:
- Many tenants can share one IP. Corporate NAT gateways, VPNs and outbound proxies put large organizations behind a few addresses. An IP limit strict enough to stop abuse will throttle legitimate users who happen to share that egress.
- One tenant can use many IPs. A customer's integration running on autoscaled workers or serverless functions spreads its traffic across many addresses, so each IP looks harmless while the tenant as a whole overloads you.
- The expensive work happens after authentication. An edge proxy sees a cheap HTTP request. It cannot tell a cached read from a report that scans millions of rows, because that depends on who the caller is and what they asked for.
The limiter therefore belongs at the monolith's tenant-resolution boundary: the point right after the tenant is identified from the API token or session, and before any controller or job runs. This approach suits teams running a shared multi-tenant codebase who want fairness between tenants without splitting the application into separate services.
Tenant A users ─┐ Tenant C integration
Tenant B users ─┼─► Corp NAT (1 IP) (autoscaled, many IPs)
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌──────────────────────────────────────────────────────────────┐
│ Edge proxy / IP rate limit │
│ sees only: source IP + a cheap-looking HTTP request │
│ A+B share 1 IP -> throttled together; C spread -> passes │
└──────────────────────────────┬───────────────────────────────┘
▼
┌──────────────────────── Monolith ────────────────────────────┐
│ ┌──────────────────────────────┐ │
│ │ Auth: token/session -> tenant│ │
│ └──────────────┬───────────────┘ │
│ ▼ ┌────────────────┐ │
│ ┌──────────────────────────────┐ take │ Redis │ │
│ │ Per-tenant limiter │◄──────►│ bucket:A │ │
│ │ (tenant-resolution boundary) │ token │ bucket:B │ │
│ └──────┬───────────────┬───────┘ │ bucket:C │ │
│ allow │ │ deny └────────────────┘ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Controllers │ │ 429 Too Many │ │
│ │ and jobs │ │ Requests │ │
│ └──────┬───────┘ └──────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Shared: DB pool | worker threads | cache │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
IP-based edge limits throttle tenants that share a NAT address and miss tenants spread across many IPs, so the per-tenant token-bucket check runs inside the monolith right after the tenant is resolved and before any controller or job touches shared capacity.
Who should use per tenant rate limiting in a monolith?
This design is aimed at teams that run one shared application, usually a monolith, serving many customer accounts from the same database, worker pool and cache. It fits best when these conditions hold:
- Tenants differ widely in size or behavior. If a few accounts run bulk imports, heavy reports or aggressive integrations while most do light interactive work, a per-tenant cap stops the few from degrading the many.
- You already run Redis, or can run it reliably. The limiter needs one shared store that every app instance reaches with low latency. If adding Redis means taking on a new operational dependency you cannot support, weigh that cost first.
- Tenant identity is known after authentication. The limiter keys on the tenant resolved from a token or session. Applications whose traffic is mostly anonymous get more from edge or IP limits.
- You run more than one app instance. With a single instance, an in-process limiter gives the same result without a network hop.
- You sell plans with different entitlements. Per-plan defaults and per-tenant overrides map directly onto pricing tiers and contract exceptions.
It is a poor fit for single-tenant deployments, for systems where one tenant is the entire workload, and for teams whose real problem is total capacity rather than fairness between tenants. The trade-offs section below covers those cases in more detail.
How does per tenant rate limiting work end to end?
The request path has five steps, each covered in detail in later sections:
- Resolve the tenant. Authentication middleware maps the API token or session to a tenant ID. The limiter only trusts this value, never a client-supplied header.
- Classify the operation. The route or job type maps to an operation class such as
read,write,searchorexport, along with a token cost that reflects how expensive the work is. - Look up the limits. The limiter combines the tenant's plan defaults with any per-tenant overrides to get a capacity and refill rate for that class, cached in-process for a short time.
- Take tokens atomically in Redis. A Lua script reads the bucket for
rl:{tenant_id}:{op_class}, refills it based on elapsed time measured by the Redis clock, and either subtracts the cost or reports how long to wait. Because the script runs atomically, concurrent requests from many app instances cannot double-spend the same tokens. - Allow or reject. On allow, the request continues to controllers or the job runs. On deny, the API returns 429 with
Retry-Afterand a fixed JSON body, and background workers re-enqueue the job with a delay.
Alongside this path, per-tenant concurrency caps and timeouts bound how long admitted work can hold shared resources, and every decision emits a metric or log line so limits can be tuned from real traffic.
How does a token bucket work compared with fixed and sliding windows?
A token bucket has two parameters: capacity, the most tokens the bucket can hold, and refill rate, how many tokens are added per second. Each request spends one token and is rejected when the bucket is empty. Nothing runs on a timer. The bucket stores only its token count and the timestamp of its last update. On each request, the limiter computes elapsed × rate, adds that many tokens, caps the total at capacity, and then tries to spend one token. This is called lazy refill. Because of it, idle tenants cost nothing until their next request.
Capacity sets the largest burst a tenant can send. Refill rate sets the long-run ceiling. A tenant that has been quiet can spend its saved tokens all at once, but after that it can sustain only the refill rate.
Worked example (illustrative estimate): assume capacity 20 and refill 5 tokens/second. An idle tenant fires 20 requests at once, and all 20 succeed. A request 1.2 seconds later finds 6 tokens available (1.2 × 5). Over any 60-second span, the most the tenant can get through is 20 + 5 × 60 = 320 requests.
| Approach | Bursts | Window-boundary spikes | State per key | Effort |
|---|---|---|---|---|
| Fixed window counter | Uncontrolled within window | Up to 2× limit across a boundary | One counter | Lowest |
| Sliding window log | Strictly bounded | None | One timestamp per request | High memory |
| Sliding window counter | Approximately bounded | Smoothed, approximate | Two counters | Moderate |
| Token bucket | Explicit, tunable | None | Two values | Moderate |
We use the token bucket because it makes burst size an explicit setting instead of a side effect of where window boundaries fall. It also keeps per-tenant state constant in size, and its read-compute-write cycle fits in a single atomic Redis script.
How should you design rate limit keys by tenant and operation class?
A single bucket per tenant still lets one workload starve another inside that tenant. If a nightly report job and a support agent's dashboard draw from the same budget, the report can drain it, and the agent gets 429s for a job they never started. The fix is to key buckets by tenant and operation class, for example rl:{tenant_id}:{op_class}. Each class then has its own capacity and refill rate, and exhausting export leaves read untouched.
Keep the set of classes small and tie each class to its backend cost, not to individual routes:
| Operation class | Example endpoints | Tokens per request (illustrative) | Why it is separate |
|---|---|---|---|
read |
list, get by ID | 1 | High volume, cheap, latency-sensitive |
write |
create, update | 3 | Locks, index updates, triggers |
search |
full-text, filtered queries | 5 | Unpredictable query plans |
export |
CSV, report generation | 20 | Long scans, large result sets |
Within a class, weight each request by cost instead of counting every call as one. A filtered search across a year of data should cost more tokens than one across a day. With weighting, the limit tracks database load, which is the resource you are protecting, rather than raw request count.
Resolve limits in two layers: per-plan defaults, then per-tenant overrides stored in configuration or on the tenant record. Cache the resolved values in-process with a short TTL so you avoid a lookup on every request. Because the values come from data, you can raise a limit for one customer or tighten a misbehaving one without a deploy, and the change takes effect within one TTL.
import time
PLAN_DEFAULTS = {
"starter": {"read": (100, 10), "write": (60, 3), "search": (50, 2), "export": (40, 0.2)},
"enterprise": {"read": (1000, 100), "write": (600, 30), "search": (500, 20), "export": (400, 2)},
}
BASE_COST = {"read": 1, "write": 3, "search": 5, "export": 20}
CACHE_TTL_S = 30 # an override takes effect within one TTL, no deploy needed
_cache = {} # tenant_id -> (expires_at, limits)
def resolve_limits(tenant):
"""Plan defaults merged with per-tenant overrides, cached in-process."""
hit = _cache.get(tenant.id)
if hit and hit[0] > time.monotonic():
return hit[1]
limits = dict(PLAN_DEFAULTS[tenant.plan])
# Overrides live on the tenant record, e.g. {"export": [200, 1]}
for op_class, (cap, rate) in (tenant.rate_limit_overrides or {}).items():
limits[op_class] = (cap, rate)
_cache[tenant.id] = (time.monotonic() + CACHE_TTL_S, limits)
return limits
def bucket_key(tenant_id, op_class):
# One bucket per tenant per class: draining "export" leaves "read" untouched.
return f"rl:{tenant_id}:{op_class}"
def request_cost(op_class, days_spanned=1):
# Weight within a class so the limit tracks DB load, not request count:
# a year-long search costs more tokens than a one-day search.
cost = BASE_COST[op_class]
if op_class == "search":
cost += min(days_spanned // 30, 12)
return cost
def check(tenant, op_class, take_tokens, **kw):
capacity, refill = resolve_limits(tenant)[op_class]
# take_tokens wraps the atomic token-bucket Lua script executed in Redis.
return take_tokens(bucket_key(tenant.id, op_class), capacity, refill,
request_cost(op_class, **kw))
Builds a per-tenant, per-operation-class bucket key, charges cost-weighted tokens, and resolves limits from plan defaults plus tenant overrides cached in-process with a short TTL.
How do you implement an atomic token bucket in Redis with a Lua script?
The script
The Lua script takes one key, the bucket hash for a tenant and operation class, plus three arguments: capacity, refill rate in tokens per second, and the cost of this request. It calls TIME first and builds a millisecond timestamp from the seconds and microseconds it returns. It then reads tokens and ts with HMGET. A missing hash means a new or expired tenant, so it starts with a full bucket. It adds elapsed × rate tokens, capped at capacity, and allows the request if the balance covers the cost.
On allow, it subtracts the cost. On deny, it computes the wait as (cost − tokens) / rate. In both cases it writes the state back with HSET and sets PEXPIRE to roughly the time needed to refill from empty. After that point an idle bucket is indistinguishable from a full one, so letting it expire loses nothing.
Return integers, such as an allowed flag, remaining tokens, and retry-after in milliseconds. Redis truncates Lua numbers to integers when it converts a reply, so a fractional wait would be silently rounded down.
The script uses TIME because every monolith instance writes to the same bucket. If each instance passed its own clock, a host running a few seconds fast would mint extra tokens, and one running slow would starve the tenant. Redis is the single clock all instances share.
The middleware
Resolve the tenant from the authenticated principal, never from a client-supplied header. Map the route to its operation class and cost, then call the script with EVALSHA, reloading it if Redis replies NOSCRIPT.
On deny, return 429. Set Retry-After in whole seconds, rounded up, and send a fixed JSON body with an error code, the operation class, and retry_after_ms. SDKs can then parse the response and back off without guessing.
import math
import redis
TOKEN_BUCKET_LUA = """
local t = redis.call('TIME') -- one shared clock for every app instance
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local cap, rate, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3])
local s = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(s[1]) or cap -- missing hash: new or expired, so full
local ts = tonumber(s[2]) or now
tokens = math.min(cap, tokens + math.max(0, now - ts) * rate / 1000)
local allowed, wait_ms = 0, 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
else
wait_ms = math.ceil((cost - tokens) * 1000 / rate) -- round up before integer reply
end
redis.call('HSET', KEYS[1], 'tokens', tostring(tokens), 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap * 1000 / rate)) -- idle = full, safe to drop
return {allowed, math.floor(tokens), wait_ms}
"""
r = redis.Redis(host="redis.internal", port=6379)
token_bucket = r.register_script(TOKEN_BUCKET_LUA)
def check_rate_limit(tenant_id, op_class, cost, capacity, rate):
# tenant_id must come from the authenticated principal, not a request header
allowed, remaining, wait_ms = token_bucket(
keys=[f"rl:{tenant_id}:{op_class}"], args=[capacity, rate, cost])
if allowed:
return None
headers = {"Retry-After": str(math.ceil(wait_ms / 1000))} # whole seconds, rounded up
body = {"error": "rate_limited", "operation_class": op_class,
"retry_after_ms": wait_ms} # fixed shape that SDKs parse for backoff
return 429, headers, body
A Lua token bucket that reads the clock with Redis TIME, called from a Python helper through redis-py's register_script (which handles EVALSHA and NOSCRIPT reloads), and the 429 response with Retry-After and a fixed JSON body that the middleware returns.
How do you apply the same limits to background jobs and long-running work?
Background work is where a single tenant does the most damage, because nobody waits on the response and the job queue absorbs whatever arrives. Use the same Redis buckets and the same key scheme, checked at one of two points.
- At enqueue, when a tenant action creates the job, such as a bulk import or a report. A denial here can surface as the same 429 the API already returns, so the caller learns immediately.
- At dequeue, when jobs come from schedules, webhooks or fan-out with no caller to reject. If the worker cannot take a token, it re-enqueues the job with a delay equal to the bucket's retry hint. It should not spin, and it should not drop the job.
Arrival is not consumption
A token bucket limits how often work starts. It does not limit how long that work runs. One allowed request can trigger a full table scan, and one allowed job can run for twenty minutes. Three controls cover what the bucket cannot:
- Per-tenant concurrency caps. Keep a Redis counter or lease set per tenant and job class. Acquire a slot before running and release it afterward, with a TTL so a crashed worker cannot leak the slot. This limits how many workers one tenant can hold at once.
- Query and job timeouts. Set statement timeouts on the database session and a wall-clock deadline on each job. Any runaway work then has a known upper bound.
- Cost weights. Charge tokens in proportion to the expected cost of the work. For example, an export of 50,000 rows might consume 50 tokens while a single-record sync consumes 1. Without weights, the bucket treats a heavy request the same as a cheap one.
import time, uuid
import redis
r = redis.Redis(host="localhost", port=6379)
MAX_CONCURRENT = 2 # workers one tenant may hold per job class
LEASE_TTL = 1800 # seconds; longer than the job deadline below
def acquire_slot(key):
# Lease set: member = lease id, score = expiry time.
now = time.time()
r.zremrangebyscore(key, "-inf", now) # reclaim leases from crashed workers
lease = str(uuid.uuid4())
r.zadd(key, {lease: now + LEASE_TTL})
r.expire(key, LEASE_TTL)
if r.zcard(key) > MAX_CONCURRENT: # over the cap: back out
r.zrem(key, lease)
return None
return lease
def handle(job, queue, db):
# take_tokens: the same Lua token bucket the API uses (defined elsewhere).
# Cost weight, e.g. 50 for a 50,000-row export, 1 for a single-record sync.
allowed, retry_after = take_tokens(job.tenant_id, cost=job.cost_weight)
if not allowed:
queue.enqueue_in(retry_after, job) # delay, don't spin or drop
return
key = f"conc:{job.tenant_id}:{job.job_class}"
lease = acquire_slot(key)
if lease is None:
queue.enqueue_in(5, job) # tenant already at its cap
return
try:
with db.cursor() as cur:
cur.execute("SET statement_timeout = '30s'") # bound each query
job.run(deadline=time.monotonic() + 900) # bound the whole job
finally:
r.zrem(key, lease) # release even on failure
A worker checks a weighted token bucket at dequeue, then takes a per-tenant concurrency lease with a TTL and runs the job under query and wall-clock timeouts, re-enqueuing with a delay when either limit says no.
How do you roll out and observe per tenant limits safely?
Treat a new limit as a hypothesis until production traffic confirms it. Every decision the limiter makes should emit a counter labeled with tenant, operation class and outcome (allowed or rejected). If your metrics backend struggles with one series per tenant, keep per-tenant detail in structured logs and aggregate the metrics by plan tier. These counts answer the questions you will be asked: who is hitting the limit, on which operation, and how often.
Shadow mode first
Deploy each new limit in log-only mode. The script still runs and the bucket still drains, but a request that would be rejected is logged and allowed through. After a week or two of real traffic you can see which tenants would have been throttled and whether those tenants are abusive or simply large customers you underestimated. Limits set this way come from data rather than guesswork. That means fewer retunes and fewer support escalations from customers who were blocked during normal work.
Alert on persistent pressure
A single rejection is not a problem. A tenant that sits at or near zero tokens for hours is a problem. It is either a noisy neighbor or a customer who has outgrown its plan. Alert on that sustained pattern, not on individual rejections.
Keep overrides in config
Support and sales will ask to raise limits for particular tenants, often urgently. Store per-tenant overrides in configuration that can be changed without a deploy, and audit every change. A wrong value does more than throttle the tenant. To that tenant it looks like an outage, so a fix that needs a code change or hotfix extends the outage.
How can per tenant rate limiting save engineering time?
The main saving is in incident work. Without per-tenant limits, a noisy neighbor shows up as a platform-wide latency problem. On-call engineers have to work out which tenant caused it, then contain it by hand, often by blocking an API token, killing jobs or scaling the database during the incident. With limits in place, the offending tenant is throttled automatically, the other tenants stay healthy, and the limiter's metrics already name the tenant and operation class involved. The investigation starts from an answer instead of a guess.
It also saves time in several quieter ways:
- Fewer one-off defenses. Without a shared mechanism, teams tend to add ad hoc guards to individual endpoints and jobs: a hard-coded cap here, a special-case sleep there. One limiter with operation classes replaces those, so there is one place to read, test and change.
- Customer-specific changes without deploys. Because limits come from plan defaults and per-tenant overrides in data, raising a limit for a contract or tightening a misbehaving integration is a configuration change, not a code change, review and release.
- Predictable client behavior. A consistent 429 with
Retry-Afterand a fixed JSON body lets SDKs and integrations back off on their own. That reduces support tickets asking why requests failed and how long to wait. - Capacity planning from data. Per-tenant, per-class counters show which customers consume which kinds of capacity, which helps when deciding whether to scale the database, adjust plan tiers or move a workload elsewhere.
- Delaying a service split. Containing heavy tenants inside the monolith can remove the pressure to extract services purely for isolation, which is a much larger engineering project.
Worked example (illustrative estimate): assume a team has two noisy-neighbor incidents a month, each taking 3 engineer-hours to diagnose and mitigate across on-call and follow-up, and that limits plus per-tenant metrics cut that to 1 engineer-hour per incident. The saving is 2 × (3 − 1) = 4 engineer-hours a month, before counting fewer ad hoc fixes. These inputs are assumed, not measured. Substitute your own incident counts and durations, and set the result against the ongoing tuning and support cost described below.
What are the trade-offs, and when should you avoid this approach?
Every request now pays a Redis round trip. Redis also becomes a dependency for every tenant, so you must decide what happens when it is slow or down:
- Fail-open keeps serving traffic but drops all protection at the moment the system is already unhealthy.
- Fail-closed turns a Redis outage into a full application outage.
- A local in-process fallback bucket, sized at roughly the tenant limit divided by the instance count, keeps approximate protection. It drifts when autoscaling changes the instance count.
Very large tenants create hot keys, because each bucket lives on a single Redis Cluster shard. Splitting a tenant's bucket into sub-buckets spreads that load. The cost is accuracy: each sub-bucket sees only part of the traffic, so the tenant gets throttled early or overshoots its limit. Cost weights also drift as code and data change, so an endpoint priced at 5 tokens can quietly become ten times more expensive. Generous bursts let many tenants spike at the same moment.
Per-tenant limits do nothing when many well-behaved tenants together exceed capacity. You still need a global limit or a load-shedding layer.
The initial build is modest. The ongoing cost is in tuning, the config surface and support requests about throttling.
| Alternative | Fits better when |
|---|---|
| Edge or gateway limits | Abuse is anonymous or IP-based, or you only need coarse caps |
| In-process limiters | You run a single instance or a single-tenant deployment |
| Per-tenant queues | The work is asynchronous and fairness matters more than rejecting requests |
| Separate services for heavy tenants or workloads | One tenant or workload dominates, or there is no shared hot resource to protect |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to per tenant rate limiting, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.