Why do per-replica rate limiters fail under autoscaling?

Distributed rate limiting enforces one limit across every instance of a service, rather than a separate limit inside each process. You need it when the limit protects something shared: a downstream API with a contract quota, a database with a fixed connection budget, or a per-customer allowance that should not depend on which pod answered the request.

An in-memory limiter counts only the traffic its own process sees. If each replica allows 100 requests per second, the service allows N × 100. Take an illustrative estimate with assumed inputs: 4 replicas give 400 rps, and an autoscaler that scales out to 12 under load raises the ceiling to 1,200 rps. The limit grows exactly when the downstream system is most likely to be under pressure.

The per-pod counts also depend on how the load balancer spreads traffic. With long-lived HTTP/2 or gRPC connections, a few heavy clients can stick to one pod. That pod returns 429s while its neighbors have spare budget. A single tenant can be throttled below its quota on one replica and allowed above it on another.

Scaling events move the effective limit without anyone changing configuration. Scale-in tightens it, scale-out loosens it, and pod restarts reset the counters.

Envoy's local rate limit filter has the same problem. Each sidecar or gateway instance keeps its own token bucket, so the configured value is multiplied by the number of Envoy instances.

None of these problems come from the algorithm. Token buckets work fine. The decision that matters is where the shared counter lives and which component owns it: your application talking to Redis, or Envoy calling out to a global rate limit service.

How does a token bucket in Redis work across replicas?

Each limited key, such as a user ID or API key, maps to a small Redis hash holding two fields: the current token count and the timestamp of the last refill. Nothing refills tokens in the background. On each request, the limiter computes how much time has passed since the stored timestamp, adds elapsed × rate tokens up to the bucket's capacity, subtracts one if enough tokens remain, and writes both fields back. Idle keys therefore cost nothing until they are touched again.

This read-modify-write must be atomic. With a plain GET followed by a SET, two replicas can both read "1 token left", both decide to admit, and both write 0. The bucket has admitted two requests against one token. Under load, many replicas hit the same key concurrently, so these races are constant and the effective limit drifts well above the configured one. A Lua script fixes this because Redis runs each script to completion without interleaving other commands.

The script should take the current time from Redis's TIME command rather than from a timestamp passed in by the caller. Replica clocks disagree. A replica running ahead can mint tokens early, and one running behind can compute negative elapsed time.

In Redis Cluster, every key a script touches must hash to the same slot. If you use multiple keys per limit, wrap the shared part in a hash tag like {tenant:42}. Hash tags pin keys to one shard, so a few very busy tenants can concentrate load on a single node and create a hot shard.

Fixed-window counters are simpler but leakier. A client can spend the full limit at the end of one window and again at the start of the next, which admits up to twice the limit within a short span.

import redis

# Runs atomically on the Redis server, so concurrent replicas cannot interleave.
TOKEN_BUCKET = """
local key      = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])  -- tokens added per second

-- Use the Redis server clock, not the caller's, so replica clock skew is irrelevant.
local t   = redis.call('TIME')
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local state  = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity  -- new keys start full
local ts     = tonumber(state[2]) or now

-- Lazy refill: credit the time elapsed since the last request, capped at capacity.
local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = 0
if tokens >= 1 then
  tokens = tokens - 1
  allowed = 1
end

redis.call('HSET', key, 'tokens', tokens, 'ts', now)
-- Idle buckets refill to full anyway, so let them expire.
redis.call('EXPIRE', key, math.ceil(capacity / rate) * 2)
return allowed
"""

r = redis.Redis(host="localhost", port=6379)
take_token = r.register_script(TOKEN_BUCKET)

# The {tenant:42} hash tag keeps every key for this tenant in one Cluster slot.
allowed = take_token(keys=["ratelimit:{tenant:42}"], args=[10, 5])
print("admit" if allowed == 1 else "reject")

A Lua token bucket that refills lazily, reads the clock from Redis TIME, and does the whole read-modify-write atomically, called from Python against a hash-tagged key.

How does Envoy rate limiting work in local and global modes?

Envoy offers two separate mechanisms, and they solve different problems.

Local rate limiting runs a token bucket inside each Envoy instance. You configure a bucket size, a refill amount, and a refill interval, and the proxy enforces them in memory with no external dependency. A local limit is fast and cannot fail because of a remote outage. It has the same weakness as any per-replica limiter, though: the effective limit grows with the number of proxies. It works well as a coarse safety valve and is unsuitable for a contractual per-tenant quota.

Global rate limiting adds a network call. On each route you define rate limit actions that build a descriptor from request attributes, such as a request header, the remote address, or a fixed generic key. Envoy sends these descriptors over gRPC to an external rate limit service. That service matches them against configured limits and returns OK or OVER_LIMIT. The reference implementation stores its counters in Redis. This puts another hop in the request path, and it adds a service your team has to deploy, scale, configure, and alert on alongside Redis.

Descriptors can only use data the proxy sees on the request. Suppose the limit depends on a tenant's plan tier stored in a database. Then something upstream has to resolve the tier and attach it as a header first, or the rate limit service has to look it up itself. Both options move business logic into infrastructure.

That workaround also creates a spoofing risk. If an auth layer injects a trusted header such as a tenant ID, the edge must strip any client-supplied copy of it. Otherwise a client can pick its own bucket. The remote address has a similar problem: it is only trustworthy if Envoy is configured correctly to decide which forwarded-for hops to trust.

# Fragment of an HTTP connection manager config (Envoy v3 API)
use_remote_address: true       # derive the client address from the connection itself
xff_num_trusted_hops: 0        # trust no client-supplied X-Forwarded-For hops
route_config:
  virtual_hosts:
  - name: api
    domains: ["*"]
    routes:
    - match: { prefix: "/api/" }
      route:
        cluster: backend
        rate_limits:
        # Descriptor 1: per-tenant bucket keyed on a header.
        # Safe only if the edge strips client copies of x-tenant-id
        # before a trusted auth layer sets it.
        - actions:
          - request_headers:
              header_name: x-tenant-id
              descriptor_key: tenant_id
        # Descriptor 2: per-client-IP bucket.
        - actions:
          - remote_address: {}
        # Descriptor 3: one shared bucket for the whole route.
        - actions:
          - generic_key:
              descriptor_value: api_global
http_filters:
- name: envoy.filters.http.ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
    domain: api                  # must match the domain in the rate limit service config
    timeout: 0.05s               # the extra hop is on the request path; keep it bounded
    failure_mode_deny: false     # fail open if the rate limit service is unreachable
    rate_limit_service:
      transport_api_version: V3
      grpc_service:
        envoy_grpc:
          cluster_name: ratelimit  # separate service to deploy, scale and monitor
- name: envoy.filters.http.router
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

An Envoy global rate limiting config: route-level actions build descriptors from a header, the remote address and a generic key, and the ratelimit filter sends them over gRPC to an external rate limit service.

Redis token bucket vs Envoy global rate limit: which should you use?

The three options differ less in algorithm than in where the decision is made and what the decision can see.

Criterion In-app Redis bucket Envoy local limit Envoy global limit
Enforces a true global quota Yes No, per proxy instance Yes
Precision High, exact per key Drifts as replicas scale High, bounded by counter granularity
Business awareness Full: tenant, plan, cost per call Route and header level only Descriptors built from request attributes
Enforcement point Inside the service, after parsing Sidecar or edge proxy Proxy, via an external rate limit service
Added latency One Redis round trip None beyond proxy work One gRPC call, plus its datastore call
Ownership Product team Platform team Platform team runs the service and its store
Duplication Re-implemented per language and service None None
Config complexity Code and tests Small proxy config Descriptors in proxy config plus rules in the service

In-app Redis suits product teams whose quotas depend on data only the application has, such as a tenant's plan, a per-endpoint cost weighting, or a billing state loaded from the database. The proxy never sees those values unless you copy them into headers, and that brings back the trust problem described above. The price is a client library per language and the risk that each one rounds or refills differently.

Envoy global limits suit platform teams that already run a mesh. One service and one config enforce consistent limits for every workload, whatever its language.

A token bucket is not mandatory in either setup. Fixed-window counters are cheaper, a single increment with an expiry, but they allow up to twice the limit across a window boundary. Sliding-window counters smooth out that burst at the cost of reading two windows per check. Some common global rate limit service implementations count per fixed time unit, so check which algorithm your deployment uses before relying on burst behavior.

How do you layer local, global, and in-app limits?

In practice, three layers work well together, and each one owns a distinct decision.

  1. Envoy local rate limit (per proxy instance). This layer absorbs bursts and floods with an in-memory token bucket. It makes no network call, so it can safely reject abusive traffic before that traffic reaches anything shared. Set it loosely, a little above what any single replica should ever see. Its job is to protect the global layer and the service, not to enforce customer quotas.
  2. One global mechanism. This is either Envoy's global rate limit service or a Redis token bucket, and it owns shared quotas such as "1,000 requests per minute per API key across all replicas." Choose one. If two independent counters enforce the same quota, they will drift, and the limits will be impossible to explain to anyone who asks.
  3. In-app checks. Keep these for rules that need application context the proxy cannot see. Examples are "three password resets per account per hour" or limits tied to plan tiers stored in your database.

A request flows through them in that order: local check, then global descriptor check, then business logic. Each rejection happens at the cheapest layer that has enough information to make the decision.

Centralizing the first two layers in the proxy pays off in engineering time. Without it, every service team writes its own limiter in its own language, each with slightly different key formats, failure behavior, and headers. With it, quota policy lives in proxy configuration. Teams can change it without redeploying services, and platform engineers maintain one implementation and one set of dashboards instead of one per runtime.

  Client request
        |
        v
+--------------------------------------+
| Envoy proxy (one per replica)        |
|                                      |
| [1] Local token bucket, in-memory    |--> 429 burst / flood
|     no network call, set loosely     |
|               |                      |
|               v                      |    +--------------------------+
| [2] Global descriptor check          |<-->| Global limiter (pick ONE)|
|     shared quota across replicas     |    | Envoy global RLS, or     |
|     over quota --> 429               |    | Redis token bucket       |
|               |                      |    +--------------------------+
+---------------|----------------------+
                |
                v
+--------------------------------------+
| Service (any language)               |
|                                      |
| [3] In-app check, needs app context  |--> 429 business rule
|     e.g. 3 password resets/acct/hr,  |
|     plan-tier limits from database   |
|               |                      |
|               v                      |
|     Business logic                   |
+--------------------------------------+

 Layers 1-2 live in proxy config: one implementation for all services.

A request passes the in-memory local limit, then one shared global quota check, then app-context business rules, and each layer rejects only what it alone has the information to decide.

What does a limiter call per request cost in latency and Redis load?

Whichever option holds the counter, each limiter call is a network hop on the request path, so price it before you commit.

Illustrative estimate (assumed numbers, not a benchmark). Assume an intra-zone Redis round trip of 0.5 ms and 20,000 requests per second across 40 replicas. The token bucket is a single atomic script, so each check is one Redis command.

  • Added latency per request ≈ RTT + script execution ≈ 0.5 ms at the median. Tail latency matters more, because a slow Redis shard delays every request it serves.
  • Redis ops/s = RPS × calls per request = 20,000 × 1 = 20,000 ops/s.

To cut that load, each replica can lease N tokens at once and spend them from memory. Redis ops/s then becomes RPS ÷ N. The cost is accuracy. Leased tokens that have not been spent are invisible to other replicas, so the worst-case overshoot is roughly replicas × N.

Approach Redis ops/s (assumed inputs) Added latency Worst-case over-admission
One call per request 20,000 ~RTT on every request ≈ 0
Lease N = 50 400 ~RTT once per 50 requests ~40 × 50 = 2,000 requests
Local cache, sync every 100 ms 40 × 10 = 400 ~0 on the hot path Traffic admitted during one sync interval

Multi-region deployments make this harder. A cross-region round trip costs tens of milliseconds, which is too slow to pay on every request. Teams usually take one of two approaches:

  • Split the limit into per-region quotas. For example, give each of two regions 50% of the global limit. Each region then enforces its share locally.
  • Enforce locally and reconcile counts asynchronously. This accepts some overshoot during the sync window in exchange for keeping latency low.

When should you fail open or fail closed, and how do you report limits?

Before choosing a failure policy, decide whether you need a strict global limiter at all. If the limit exists to protect your own capacity, per-replica local limits sized for the current replica count usually do the job without a network dependency. Strict global enforcement is worth its cost when the limit protects a shared downstream system, such as a third-party API quota or a database, or when it enforces a contractual per-tenant quota.

Choosing a failure policy

When Redis or the rate limit service is unreachable, every request needs a decision:

  • Fail open keeps traffic flowing, but it removes protection at the moment something is already wrong.
  • Fail closed turns a limiter outage into a full service outage.

A practical middle ground is to fail open to the local limiter. The global check is skipped, and the coarse per-replica limit still caps load.

Envoy's global rate limit filter allows requests through by default when the rate limit service errors or times out. Treat this as a decision to make, not a default to inherit. Set the filter's failure-mode behavior and its timeout explicitly.

Reporting limits to clients

Return HTTP 429 when a request is limited. Add Retry-After whenever you can compute a wait. For a token bucket, that wait is the time until the next token is available. Without it, clients retry immediately and amplify the load. Never return 429 for a limiter error. A 429 tells the client it exceeded its quota, so returning one for an internal failure misdirects the client.

Observability

Emit counters for allowed, limited, and errored decisions, labeled by limit rule. Per-key labels can explode metric cardinality, so track the top limited keys separately. Alert on:

  • Errored decisions: any sustained rate means you are silently failing open or closed.
  • Hot keys: a single tenant dominating the limited count.
# http_filters for an HCM: local limiter first as a coarse per-replica cap,
# then the global filter. If the rate limit service is down, the global
# check fails open but the local bucket still bounds load on this replica.
http_filters:
- name: envoy.filters.http.local_ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
    stat_prefix: http_local_rate_limiter
    token_bucket:              # size for (global limit / replica count) plus headroom
      max_tokens: 200
      tokens_per_fill: 200
      fill_interval: 1s
    filter_enabled:
      runtime_key: local_rate_limit_enabled
      default_value: {numerator: 100, denominator: HUNDRED}
    filter_enforced:
      runtime_key: local_rate_limit_enforced
      default_value: {numerator: 100, denominator: HUNDRED}
- name: envoy.filters.http.ratelimit
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
    domain: api_gateway
    # Stated explicitly even though false is the default: on error or
    # timeout, allow the request rather than turning a limiter outage
    # into a full outage. Set true only if overload is worse than downtime.
    failure_mode_deny: false
    # Bound the latency the limiter can add to every request.
    timeout: 0.05s
    # Add X-RateLimit-* headers to responses so clients can pace themselves.
    enable_x_ratelimit_headers: DRAFT_VERSION_03
    rate_limit_service:
      transport_api_version: V3
      grpc_service:
        envoy_grpc:
          cluster_name: rate_limit_service
# Descriptors still come from rate_limits actions on routes/virtual hosts.
- name: envoy.filters.http.router
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

An Envoy filter chain that sets the global rate limit filter's failure mode and timeout explicitly and places a per-replica local token bucket in front of it, so that load stays capped when the rate limit service is unreachable.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed rate limiting, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.