Service to Service Rate Limiting with Token Buckets and Quotas

What service to service rate limiting is and why internal API limits differ

Service to service rate limiting caps how many requests one internal service may send to another over a period of time. Inside a company it works as a capacity contract between teams. The owner of a shared service states how much load each consumer can depend on, and the limiter holds both sides to that number. Stopping abuse is not the goal, because the callers are colleagues.

Internal API limits differ from public edge limits in two ways. First, every caller is known and authenticated through a service identity, mTLS certificate or signed token. You can therefore key limits on who is calling instead of on IP addresses, and you can talk to the caller's team when a limit is hit. Second, the traffic shape is different. Batch jobs start on the hour. A single user request fans out into dozens of downstream calls. Retries multiply load at exactly the moment a dependency is struggling. Internal traffic tends to be legitimate but very bursty, so a limit has to tolerate short spikes and still bound sustained load.

Who should use it

The primary audience is the team that owns a shared internal service with several internal callers. That team knows the service's capacity and is paged when it degrades, so it should set and enforce the quotas. Platform teams that run the service mesh, sidecars or internal gateways are the second audience. They often provide the enforcement point and the identity layer that the quotas depend on.

Teams that call a rate-limited service are users too, in a different role. They need to handle 429 or RESOURCE_EXHAUSTED responses correctly, honor Retry-After, and negotiate their allocation. Without that discipline, a server-side limit tends to turn rejections into retry storms.

When to use it

Adopt it if your team runs a shared service that several internal callers depend on. It pays off once one noisy caller, such as a misconfigured backfill, can degrade latency for everyone else.

The case is strongest when these conditions hold:

• Multiple callers share one capacity pool. Isolating them is the point.
• Batch and latency-sensitive traffic mix. A reporting job and a checkout path hitting the same service need different guarantees.
• You have measured capacity. Quotas are derived from load-test numbers, so you need those numbers first.
• Callers have verified identities. mTLS or signed service tokens let you key quotas on who is calling.
• You have had, or expect, noisy-neighbor incidents. Per-caller limits and metrics make the source obvious and contain the damage.

When to avoid it

Defer it in two cases. One is a service with a single caller, where a quota protects nobody. The other is a service whose capacity you haven't measured, because any quota you set would be a guess.

Also avoid relying on it in these situations:

• The real risk is slow or expensive requests. A rate limit caps request rate, not concurrency or latency. Fifty requests per second of slow, expensive queries can still exhaust threads or connections. Use concurrency limits, timeouts and load shedding, and treat rate limiting as one layer among several. Having limits in place can create false confidence if you rely on them alone.
• Callers cannot be reliably identified. If quotas would be keyed on source IP or a self-declared header, they will be unfair or easy to evade. Put service identity in place first.
• It would replace capacity planning. A quota that rejects legitimate, expected traffic is a signal to add capacity or renegotiate, not a fix.
• The operational cost outweighs the risk. For a low-traffic service with a few well-behaved callers, maintaining quota configuration, metrics and negotiation can cost more than the incidents it prevents.

How it works

At a high level, every request passes through the same steps:

1. Identify the caller. The limiter reads a verified identity from the request, such as the subject of the caller's mTLS certificate or a claim in a signed service token. Unknown callers fall into a shared default bucket.
2. Look up the quota. Configuration maps that identity, and optionally the endpoint, to a refill rate, a burst size and a priority class.
3. Check the bucket. The limiter refills the caller's token bucket based on elapsed time and then tries to spend the request's cost in tokens. Cheap requests cost one token, and expensive ones can cost more.
4. Allow or reject. If enough tokens remain, the request proceeds. If not, the limiter returns 429 Too Many Requests or gRPC RESOURCE_EXHAUSTED, with a Retry-After value computed from the refill rate.
5. The caller backs off. A well-behaved client waits at least the hinted time, adds jitter, and caps its retries.
6. Record the decision. Allowed counts, rejected counts and bucket utilization per caller feed dashboards and alerts.

Where step 3 runs is a separate choice. It can run in process, in a sidecar or proxy, or against a central store. A global ceiling and priority classes sit above the per-caller buckets so that the combined load stays bounded. The sections below cover each piece in detail.

How the token bucket works and how to size it

A token bucket has two parameters. The refill rate is the sustained throughput a caller is entitled to, in tokens per second. The bucket size is the largest burst it can absorb before it falls back to that rate. Each request spends one token. An empty bucket means reject.

Sizing: an illustrative estimate

All inputs below are assumed:

• Tested capacity: the callee sustains 2,000 rps in load tests.
• Safety margin: 30%, which leaves a 1,400 rps budget.
• Callers: 10, with equal shares, so each gets a 140 rps refill rate.
• Peak-to-average ratio: 3, so peaks run around 420 rps.

A bucket of 280 tokens lets one caller hold 420 rps for about one second, since it drains at 420 − 140 = 280 tokens per second.

The cost shows up in the worst case. If all ten callers burst together, the callee sees about 4,200 rps for that second, which is double its tested capacity. Large buckets absorb legitimate bursts but allow correlated ones, such as a deploy or a cron job firing everywhere at once. Small buckets protect the callee but reject normal fan-out, where one user request expands into dozens of calls. Base the bucket size on how likely callers are to burst together, not on a single caller's behavior alone.

Lazy refill

No background timer is needed. Store tokens and last_refill for each caller. On each request:

1. Set tokens = min(capacity, tokens + elapsed_seconds × rate) and update last_refill. Read elapsed_seconds from a monotonic clock.
2. If tokens ≥ 1, subtract one and allow the request.
3. Otherwise, reject it and return (1 − tokens) / rate as the wait until the next token is available.

Copyimport time

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate              # sustained tokens per second (e.g. 140)
        self.capacity = capacity      # max burst in tokens (e.g. 280)
        self.tokens = capacity        # start full so the first burst is allowed
        self.last_refill = time.monotonic()  # monotonic: immune to wall-clock jumps

    def try_acquire(self):
        """Return (allowed, wait_seconds). Refills lazily; no timer thread."""
        now = time.monotonic()
        elapsed = now - self.last_refill
        # Credit tokens earned since the last call, capped at the bucket size.
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.last_refill = now

        if self.tokens >= 1:
            self.tokens -= 1
            return True, 0.0
        # Time until the fractional balance reaches one whole token.
        return False, (1 - self.tokens) / self.rate


buckets = {"checkout-svc": TokenBucket(rate=140, capacity=280)}

allowed, wait = buckets["checkout-svc"].try_acquire()
if not allowed:
    # Surface the wait to the caller, e.g. as a Retry-After hint.
    print(f"rejected, retry in {wait:.3f}s")

A single-process token bucket that refills lazily from elapsed monotonic time, rejects when empty, and reports how long the caller should wait for the next token.

Alternatives: fixed window, sliding window and leaky bucket

The token bucket is not the only option. Each alternative trades burst tolerance, memory and accuracy differently.

The fixed window allows a caller to spend its full limit at the end of one window and again at the start of the next. The sliding window log closes that gap, but its memory grows with the limit, which is expensive across thousands of keys. The sliding window counter weights the previous window's count by how much of it still overlaps the current window. That assumes requests were spread evenly, so the result is an approximation.

Internal callers are bursty for legitimate reasons: a cron job fans out, a cache expires, or a deploy restarts pods that all warm up at once. A token bucket absorbs those bursts up to a bound you choose. It still enforces the long-run rate, and it does this with constant state per key.

Use a leaky bucket when the downstream cannot absorb any burst, for example a dependency with a strict fixed rate. Callers must accept the queueing delay this adds. Use a concurrency limit when cost per request varies widely. Capping in-flight requests tracks real load better than counting arrivals does.

Designing per-client quotas keyed on caller identity

A quota is only as fair as its key. Source IP fails inside a cluster: pods share node addresses behind NAT, IPs get reused when workloads reschedule, and one IP can carry traffic from several services. A self-declared header such as X-Caller: billing is worse, because any caller can claim to be anyone. Key quotas on an identity the platform has verified, such as the subject of an mTLS certificate or the claims in a signed service token. Callers that present no identity, or one you don't recognize, share a small default quota. This keeps them working at low volume and encourages them to register.

Give each known caller its own allocation. Where operations differ sharply in cost, use per-endpoint limits or cost-weighted tokens. For example, a report export might consume 20 tokens and a lookup 1. Treat these weights as estimates and revisit them against measured latency and CPU, because the cost model will drift.

Allocations can overcommit, meaning their sum can exceed capacity, since callers rarely peak together. Back this with a global ceiling, and assign priority classes so critical-path callers are served first when that ceiling binds.

An illustrative configuration:

Every identity multiplied by every endpoint adds state, configuration and negotiation between teams. Coarser keys are cheaper to run but less fair to individual callers.

Copyquotas:
  # Callers with no identity, or one we don't recognize, share one small bucket.
  default:
    tokens_per_second: 10
    burst: 20
    shared: true
    priority: low

  callers:
    - identity: "spiffe://prod.example.internal/ns/payments/sa/checkout"
      tokens_per_second: 500
      burst: 1000
      priority: critical      # on the purchase path; served first
    - identity: "spiffe://prod.example.internal/ns/search/sa/search"
      tokens_per_second: 300
      burst: 600
      priority: standard
    - identity: "spiffe://prod.example.internal/ns/analytics/sa/reporting"
      tokens_per_second: 100
      burst: 200
      priority: batch         # can wait or retry later

  # Tokens charged per request. These weights are estimates of relative cost;
  # recheck them against measured latency and CPU, because they will drift.
  endpoint_weights:
    default: 1
    "/lookup": 1
    "/export": 20             # one export draws 20 tokens from the caller's bucket

  # Caller allocations total 900 tokens/s, above the 800 ceiling. This is
  # deliberate overcommit, since callers rarely peak at the same time.
  global_ceiling:
    tokens_per_second: 800
    # When the ceiling binds, shed lower classes before higher ones.
    shed_order: [low, batch, standard, critical]

A hypothetical quota configuration keyed on mTLS SPIFFE identities, with a shared default bucket, per-caller allocations and priority classes, cost-weighted endpoints, and an overcommitted set of allocations bounded by a global ceiling.

Where to enforce: in-process, sidecar or proxy, or central limiter

Once the keys are chosen, decide where the counter lives. There are three options, and each puts something different on the request path.

In-process. The caller's request reaches the server, middleware checks a bucket in local memory, and the handler runs or rejects. No extra hop is involved. Each replica only sees its own traffic, so a 400 req/s quota spread across eight replicas becomes 50 req/s per replica. That split holds only while the replica count stays fixed and load is spread evenly. When an autoscaler adds replicas, the effective quota grows unless each replica's share is recalculated. Sticky connections and long-lived gRPC streams cause a different problem: one replica can carry most of a caller's traffic and reject it at 50 req/s while the other seven sit idle.

Sidecar or proxy. The request passes through a local proxy before it reaches the application. Enforcement moves out of application code, but the counting model does not change. The proxy is still one instance among many unless it consults a shared store.

Central limiter. Each request triggers a call to a shared store that holds the authoritative count. This gives accurate global limits. The cost is a network hop on every request and a new dependency that every protected service relies on.

With a central store, you must decide in advance what happens when it is unreachable. Fail-open keeps traffic flowing but drops protection during the outage. Fail-closed protects the backend but turns a limiter outage into a full service outage. Choose one per endpoint and write it down.

A hybrid approach avoids most of these trade-offs. Each replica enforces a local bucket and periodically reports its consumption to the shared store. On each sync, the replica gets back an updated local allowance. The store is no longer on the hot path, and the error is limited to roughly one sync interval of traffic.

Copy(1) In-process: no extra hop; quota / replicas (400 / 8 = 50 req/s each)
          +--------------------------------------------+
Caller -->| Replica: middleware -> local bucket -> app |--> 2xx / 429
          +--------------------------------------------+

(2) Sidecar or proxy: enforcement leaves app code, counts stay local
          +--------------------------------------------+
Caller -->| Pod: sidecar proxy + bucket -> app         |--> 2xx / 429
          +--------------------------------------------+

(3) Central limiter: accurate global count, network hop on every request
          +----------------+   check/decr   +-----------------+
Caller -->| Replica / proxy|--------------->| Shared store    |
          |                |<---------------| (authoritative) |
          +----------------+  allow / deny  +-----------------+
                  |                         store unreachable?
                  v                         fail-open:   allow
             app or 429                     fail-closed: reject

(4) Hybrid: decide locally; push usage (v), pull new share (^) on a timer
   +-----------+   +-----------+   +-----------+
   | Replica A |   | Replica B |   | Replica C |   <-- requests
   +-----------+   +-----------+   +-----------+
       |   ^           |   ^           |   ^
       v   |           v   |           v   |
   +-----------------------------------------------+
   | Shared store: global usage, per-replica share |
   +-----------------------------------------------+

Request paths through in-process, sidecar or proxy, and central-store enforcement, plus a hybrid where replicas decide from local buckets and reconcile usage with a shared store off the request path.

Rejecting requests and retrying without causing a retry storm

When a caller exceeds its quota, the callee should fail fast and say so clearly. Over HTTP that means 429 Too Many Requests. Over gRPC it means RESOURCE_EXHAUSTED. Don't return a generic 500 or 503. Callers often treat those as transient faults and retry right away, which is the opposite of what you want. Include a Retry-After header with the number of seconds until the caller's bucket will plausibly have tokens again. The limiter knows its refill rate, so this value is computable rather than a guess.

On the caller side, three rules stop one rejection from turning into many:

• Honor the hint. If Retry-After is present, wait at least that long.
• Back off exponentially with jitter. Otherwise, randomize each delay across the full interval. Without jitter, every replica that was rejected at the same moment retries at the same moment, and the burst repeats.
• Cap attempts and enforce a retry budget. Limit retries per request, and also limit retries as a fraction of total traffic per client, for example 10%. The per-request cap stops one request from looping. The budget stops a fleet from multiplying its load during an incident.

Server enforcement only protects the callee. The caller still pays to build, send and serialize requests that will be rejected. A client-side token bucket sized to the caller's quota avoids that waste. So does adaptive throttling, which tracks recent requests against accepted responses and drops a growing share of requests locally as rejections rise. Treat both as an optimization. The server remains the authority, because client-side limits can be misconfigured, stale or bypassed. The retry loop below honors Retry-After, applies jittered backoff and stops at a fixed attempt cap.

Copyimport random
import time
import urllib.error
import urllib.request

MAX_ATTEMPTS = 4    # 1 initial try + 3 retries; a hard cap so a request never loops
BASE_DELAY = 0.2    # seconds; first backoff window
MAX_DELAY = 10.0    # longest single wait we accept before giving up

def retry_after_seconds(err):
    # Retry-After may be delta-seconds or an HTTP-date; this sketch only parses seconds.
    value = err.headers.get("Retry-After")
    try:
        return max(0.0, float(value)) if value else None
    except ValueError:
        return None

def fetch_with_retry(url, timeout=2.0):
    for attempt in range(MAX_ATTEMPTS):
        try:
            with urllib.request.urlopen(url, timeout=timeout) as resp:
                return resp.read()
        except urllib.error.HTTPError as err:
            # Only retry quota rejections, and stop at the attempt cap.
            if err.code != 429 or attempt == MAX_ATTEMPTS - 1:
                raise
            hint = retry_after_seconds(err)
            if hint is not None:
                if hint > MAX_DELAY:
                    raise  # server says wait longer than we are willing to; fail upward
                # Honor the hint as a floor; small jitter keeps replicas from retrying in lockstep.
                delay = hint + random.uniform(0, BASE_DELAY)
            else:
                # Full jitter: uniform over [0, capped exponential window].
                delay = random.uniform(0, min(MAX_DELAY, BASE_DELAY * 2 ** attempt))
            # A per-client retry budget (e.g. retries <= 10% of requests) would be checked here.
            time.sleep(delay)

A standard-library Python retry loop that retries only HTTP 429, treats a numeric Retry-After as a minimum wait, otherwise uses full-jitter exponential backoff, and gives up after a fixed number of attempts.

Operating quotas, pairing with other protections, and impact on cost and engineering time

Each caller needs three metrics: allowed, rejected, and bucket utilization, which is tokens consumed relative to capacity. Utilization is the leading signal because it shows a caller approaching its ceiling before any request fails. Alert on rejection that persists for several minutes, not on single spikes. Absorbing short bursts is the bucket's job.

Run every new or raised quota in shadow mode first. The limiter evaluates and logs each decision without rejecting anything, which exposes callers whose real traffic differs from what was assumed. Keep quota changes cheap: a reviewed config change listing the owner, the reason, and the expected peak. If changes are slow, teams will work around the limiter instead.

Static quotas go stale. A deployment can change per-request cost, new hardware changes capacity, and a shifted query mix changes which endpoints are expensive. Re-derive quotas from capacity tests periodically rather than treating them as permanent.

A rate limit only bounds arrivals. A caller within its rate can still exhaust workers by sending slow requests, so pair the limit with other protections:

The up-front cost is real. You need caller identity, quota configuration, capacity testing, and disciplined retries in every client. After launch, quota negotiation becomes ongoing overhead. In return, noisy-neighbor incidents become rarer. When a caller is throttled, it is clear who owns the fix. Per-caller metrics also shorten triage from "something is slow" to "this caller exceeded its quota."

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to service to service rate limiting, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.