HPA Custom Metrics Queue Depth: Scaling Workers on Backlog in Kubernetes

Why queue depth beats CPU for async workers and buffering APIs

Scaling on queue depth means the Horizontal Pod Autoscaler sets replica count from how much work is waiting in a broker, such as a RabbitMQ queue, an SQS queue, a Redis list, or Kafka consumer lag, instead of from how busy each pod's CPU is.

The reason is that most queue consumers are I/O-bound. A worker that pulls a message, calls a downstream API, and writes to a database spends most of its time waiting on the network. Each pod might sit at 15% CPU while a backlog of 200,000 messages grows behind it. A CPU-based HPA sees idle pods and does nothing. Memory is no better, because a consumer that processes one message at a time holds roughly constant memory however long the queue gets. By the time either metric moves, users are already feeling the delay, or the metric never moves at all.

This approach suits backend and platform engineers running queue consumers, background job workers, and APIs that accept a request, enqueue it, and return a 202. The test is simple: work arrives through a broker, and latency comes from time spent waiting in line, not from compute per item.

The core idea is that the signal must be backlog per replica, not raw depth. A queue of 10,000 is a crisis for 2 workers and trivial for 200. Here is an illustrative estimate with assumed inputs: if one worker drains 50 messages per second and you want any message cleared within 60 seconds, each replica can own about 3,000 messages. The HPA's job is to keep the queue at or below that per-replica share.

Who should use queue-depth autoscaling

Use it if you run workers on Kubernetes that consume from a broker and your team owns the latency of that work. Typical owners:

• Backend engineers running job workers, such as email senders, image processors, webhook dispatchers, and ETL steps, that read from RabbitMQ, SQS, Redis, or Kafka.
• Platform engineers who provide autoscaling as a shared capability and need one pattern that works across many consumer Deployments.
• API teams whose endpoints accept a request, enqueue it, and return a 202. User-visible latency for these endpoints is the time a message spends in the queue.

The approach assumes some operational groundwork. You need a way to get queue depth into the cluster's metrics APIs, either Prometheus with the Prometheus adapter or KEDA. You also need workers that can be stopped at any time without losing work. That second requirement is covered in the graceful shutdown section below. If your team cannot yet run a metrics pipeline it trusts, or your handlers are not safe to redeliver, fix those first. Autoscaling multiplies whatever behavior the workers already have.

When to use it

Scale on queue depth when most of the following are true:

• Arrival rate varies. Traffic comes in bursts, batch drops, daily peaks, or upstream retries, so a fixed replica count is either idle most of the day or too small at peak.
• Latency is queueing delay. A message waits far longer in the queue than it takes to process, and that wait is what you need to bound.
• Workers are I/O-bound. CPU and memory stay flat while the backlog grows, so resource metrics give no useful signal.
• Parallelism helps. More consumers actually drain the queue faster, meaning partitions, message groups, or ordering constraints do not cap concurrency below your planned maxReplicas.
• Downstream systems can absorb it. The databases and APIs your workers call can take the extra concurrency at peak replica count.
• Workers tolerate scale-in. Handlers are idempotent and shut down cleanly on SIGTERM.

Do not use it when load is steady, when consumers already match a Kafka topic's partition count, or when a downstream limit is the real bottleneck. These cases are covered in more detail in the alternatives section.

How it works

The mechanism is a feedback loop with four steps:

1. Measure. An exporter or the broker publishes the current backlog for the queue or consumer group as a gauge.
2. Expose. An adapter, either the Prometheus adapter or KEDA's metrics server, serves that number to Kubernetes through the external.metrics.k8s.io API, labeled so each HPA can select its own queue.
3. Compute. On every sync period, the HPA controller reads the metric. With an AverageValue target, it divides the total backlog by the current replica count and compares the result to the target. The result works out to desiredReplicas = ceil(backlog / target). The controller ignores small deviations within its tolerance, which is 10% by default. It then clamps the result to minReplicas and maxReplicas and applies the rate limits and stabilization windows from the behavior block.
4. Act. The HPA updates the Deployment's scale subresource. New pods start consuming. When the backlog falls, pods are removed and drain their in-flight work before exiting.

As replicas drain the backlog, the per-replica value drops, and the loop settles at a replica count that keeps each pod's share near the target. The following sections cover each part in detail: the metrics APIs, choosing the target, the adapter and HPA configuration, and safe scale-in.

How the HPA reads queue metrics: custom vs external metrics APIs

The HPA reads non-resource metrics through two aggregated APIs, each served by an adapter rather than by Kubernetes itself.

custom.metrics.k8s.io serves metrics tied to Kubernetes objects. A Pods metric is reported per pod and averaged. An Object metric describes a single object, such as an Ingress or a Service. external.metrics.k8s.io serves metrics about things outside the cluster, identified by name and label selectors instead of an object reference.

Broker queue depth usually belongs in the external API. The queue is not a Kubernetes object, and the backlog is shared across all consumers, not owned by any one pod. Exposing it as a Pods metric would make every pod report the same total, so the per-pod average would be meaningless. Attaching it to a Service as an Object metric works, but only by fiction. An external metric with an AverageValue target divides the total backlog by the replica count, which is the calculation you want.

The full path has six hops:

1. The broker, or an exporter in front of it, exposes queue depth.
2. Prometheus scrapes it on its scrape interval.
3. The Prometheus adapter runs a query and translates the result.
4. The adapter serves the result through the metrics API.
5. The HPA controller polls on its sync loop, which defaults to 15 seconds.
6. The HPA updates the Deployment's replica count.

Each hop adds delay and is a separate place to break. If the HPA cannot fetch the metric, it does not scale. It sets ScalingActive to false and holds the current replica count. That keeps the workload stable, but it also hides failures. A dead exporter looks the same as a quiet queue unless you alert on the HPA's conditions.

Copy +-----------------------+   hop 1: exposes queue depth
 | Broker or exporter    |   fails: exporter down, broker unreachable
 +-----------+-----------+
             | scrape (Prometheus scrape interval)
             v
 +-----------------------+   hop 2: stores the time series
 | Prometheus            |   fails: scrape errors, stale series
 +-----------+-----------+
             | PromQL query
             v
 +-----------------------+   hop 3: runs query, translates result
 | Prometheus adapter    |   fails: bad query, adapter pod down
 +-----------+-----------+
             | serves via APIService
             v
 +-----------------------+   hop 4: external.metrics.k8s.io
 | Aggregated metrics API|   (queue is not a Kubernetes object)
 +-----------+-----------+
             | poll every sync period (default 15s)
             v
 +-----------------------+   hop 5: desired = ceil(total / AverageValue)
 | HPA controller        |   metric missing -> ScalingActive=False,
 +-----------+-----------+   replicas held: dead exporter looks
             |               like a quiet queue -> alert on conditions
             | scale subresource
             v
 +-----------------------+   hop 6: replica count updated
 | Deployment (workers)  |
 +-----------------------+

The six-hop path from broker queue depth to Deployment replicas through the external metrics API; each hop adds delay and a failure point, and a missing metric freezes the replica count.

Choosing a per-replica target with AverageValue

With AverageValue, the HPA divides the metric's total by the target and rounds up: desiredReplicas = ceil(backlog / target). The target is therefore the backlog you want each pod to own. Derive it from two numbers. The first is the sustained throughput one pod achieves, measured under realistic load. The second is how many seconds of lag the business tolerates. Multiply them: target = msgs/sec per pod × acceptable lag seconds. At that ratio, the replica count the HPA picks drains the current backlog in roughly the tolerated time, ignoring new arrivals and scale-up delay.

Illustrative estimate. Assume a pod processes 20 messages/sec and 30 seconds of lag is acceptable. The target is 600. maxReplicas is 20.

Where a fixed target breaks down

• Uneven message cost. If one job type takes ten times longer than another, a single per-pod throughput figure is wrong for both. A backlog of cheap messages over-provisions. A backlog of expensive ones under-provisions. Splitting queues by job class gives each workload an accurate target.
• Partitions cap parallelism. In a Kafka consumer group, replicas beyond the partition count sit idle. Set maxReplicas no higher than the partition count.
• Downstream capacity. maxReplicas must reflect what databases and rate-limited APIs can absorb, not only what the cluster can schedule. Size connection pools to match. At 20 replicas with 10 connections each, the database must accept 200 connections plus headroom for everything else that connects to it.

Implementing it with an exporter, the Prometheus adapter, and an HPA

Start by getting queue depth into Prometheus. Many brokers expose it natively; where they don't, run a small exporter that polls the broker and publishes a gauge labeled by queue or consumer group. Use the backlog for the group your workers consume, not the topic's total size.

Next, add an external rule to the Prometheus adapter. The rule's seriesQuery discovers the series, name sets the external metric's name, and metricsQuery wraps the PromQL with <<.LabelMatchers>>. Those matchers are filled from the HPA's metric selector, so each Deployment asks for its own queue and gets only that queue's number. If you drop the placeholder or aggregate it away with a bare sum(), every HPA sees the same combined backlog.

Then define an autoscaling/v2 HPA with an External metric that names the metric, sets a selector.matchLabels for the queue, and uses an AverageValue target. With AverageValue, the HPA divides the backlog by the current replica count and compares the result to the target.

Add a behavior block. Let scale-up react quickly, for example doubling pods per period with a short or zero stabilization window. Make scale-down slower, removing a few pods per minute after a stabilization window of several minutes. Consumers should shrink slowly for two reasons. Backlog drains in bursts, so aggressive scale-in causes flapping. And in group-based consumers, each membership change triggers a rebalance that pauses consumption.

You can list a second metric, such as oldest-message age or CPU. The HPA computes a desired replica count for each metric and uses the highest.

Adapter rules are easy to get subtly wrong. Query the external metrics API directly and confirm each selector returns the value you expect before you trust the HPA.

CopyapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  minReplicas: 2
  maxReplicas: 40
  metrics:
  - type: External
    external:
      metric:
        name: queue_depth          # external metric name set by the adapter rule's `name`
        selector:
          matchLabels:
            queue: orders          # becomes <<.LabelMatchers>>, so only this queue's backlog is returned
      target:
        type: AverageValue
        averageValue: "100"        # aim for ~100 pending messages per pod
  - type: Resource                 # second metric: HPA uses the highest desired replica count
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0  # react to a growing backlog immediately
      policies:
      - type: Percent
        value: 100                   # allow doubling pods every 30s
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300  # wait out bursty drains to avoid flapping
      policies:
      - type: Pods
        value: 2                       # remove at most 2 pods/min to limit rebalances
        periodSeconds: 60

An autoscaling/v2 HPA that scales a worker Deployment on its own queue's backlog via an External AverageValue metric, adds CPU as a second metric, and uses a behavior block for fast scale-up and slow, stabilized scale-down.

Scaling in without losing work: graceful shutdown for workers

When the HPA lowers the replica count, the ReplicaSet controller picks pods to delete, and you do not control which ones. A busy worker is as likely to be chosen as an idle one. For each deleted pod, the kubelet runs any preStop hook, sends SIGTERM to the containers, and waits up to terminationGracePeriodSeconds, which defaults to 30 seconds. Any process still running after that receives SIGKILL. A worker that ignores SIGTERM keeps pulling messages until it is killed mid-job.

Handling SIGTERM

On SIGTERM, the worker should:

• Stop consuming. Cancel the subscription or exit the poll loop so the broker stops handing this pod new messages.
• Drain what it holds. Finish in-flight messages if they will complete within the grace period. Return the rest explicitly with a nack, or by releasing a lease or visibility timeout, so another replica picks them up right away instead of after a timeout expires.
• Exit cleanly. Close broker connections and exit with status zero once the in-flight count reaches zero.

Set terminationGracePeriodSeconds higher than your longest realistic job plus shutdown overhead. If a job cannot reliably finish inside any reasonable grace period, design it to checkpoint progress or to be safely abandoned instead.

Relying on broker semantics

Acknowledge a message only after its side effects are committed. Then an unacked message from a pod killed at any point, including by SIGKILL, an OOM kill, or a node failure, gets redelivered.

Redelivery means a message can be processed twice. Make handlers idempotent, for example by deduplicating on a message ID or using upserts. That makes a duplicate harmless, and it covers the failure cases graceful shutdown cannot.

Copyimport signal
import sys
import threading

stop = threading.Event()

def on_sigterm(signum, frame):
    # The kubelet sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds.
    # Only set a flag here so the current job is not interrupted.
    stop.set()

signal.signal(signal.SIGTERM, on_sigterm)

def run(client, handle):
    # 'client' is your broker SDK behind a small illustrative interface:
    # receive / ack / nack / close. Use prefetch=1 so the pod holds one message at a time.
    while not stop.is_set():
        msg = client.receive(timeout_seconds=5)  # short poll so the stop flag is seen quickly
        if msg is None:
            continue
        try:
            handle(msg)      # idempotent: dedupe on msg.id or use upserts
            client.ack(msg)  # ack only after side effects are committed
        except Exception:
            client.nack(msg, requeue=True)  # hand it back for another replica now

    # Loop exited: no new messages are pulled and the in-flight one has finished.
    # If SIGKILL lands mid-job instead, the unacked message is redelivered by the broker.
    client.close()
    sys.exit(0)

A broker-agnostic Python worker loop that stops pulling work on SIGTERM, finishes its single in-flight message, acks only after processing, nacks failures for immediate redelivery, and exits cleanly within the pod's grace period.

KEDA vs HPA with the Prometheus adapter, and other alternatives

Those properties hold whichever controller adjusts the replica count. The choice between controllers comes down to who queries the broker, whether zero replicas is allowed, and how many components you want to operate.

KEDA runs its own external metrics server. From each ScaledObject it creates and manages an ordinary HPA, so the scaling algorithm and behavior tuning work the same way. KEDA also handles the 0↔1 transition itself, which the HPA cannot. The ScaledObject below mirrors the earlier HPA, with the same target and bounds.

Scale-to-zero frees the idle pods, but the first message then waits for scheduling, image pull, and startup. Each scaler needs credentials and a network path to the broker. Avoid running KEDA and the Prometheus adapter side by side for external metrics. Typically only one service can be registered for external.metrics.k8s.io. A common split is to let the adapter serve custom metrics only and let KEDA own external metrics.

Skip queue-based autoscaling entirely in three cases:

• Steady, low-variance load. A fixed replica count is simpler and just as good.
• Partitioned queues already at full consumer count. Once consumers equal the partition count, extra replicas sit idle.
• Downstream limits. If a database or third-party API cannot absorb more concurrency, adding workers only moves the backlog downstream.

CopyapiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-worker
  namespace: workers
spec:
  scaleTargetRef:
    name: orders-worker          # same Deployment the earlier HPA targeted
  minReplicaCount: 1             # mirrors the HPA; set to 0 to allow scale-to-zero
  maxReplicaCount: 20
  pollingInterval: 15            # seconds between KEDA queries to Prometheus
  cooldownPeriod: 300            # only applies to the 1 -> 0 step when minReplicaCount is 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:                  # passed through to the HPA that KEDA creates
        scaleDown:
          stabilizationWindowSeconds: 300
  triggers:
    - type: prometheus
      metadata:
        serverAddress:
        # Same backlog query the adapter rule exposed: ready messages in the queue
        query: sum(rabbitmq_queue_messages_ready{queue="orders"})
        threshold: "30"          # target backlog per replica (AverageValue)

A minimal KEDA ScaledObject that reproduces the earlier Prometheus-adapter HPA: same Deployment, bounds, per-replica backlog target and scale-down behavior. KEDA generates the underlying HPA itself, and setting minReplicaCount to 0 is the only change needed to enable scale-to-zero.

Verifying the pipeline, and the impact on cost and engineering time

Verify each layer before trusting it

Start at the metrics API, not the HPA. kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/<ns>/<metric>" should return your queue depth. If it errors or returns nothing, the problem is in the adapter or exporter, and the HPA will not report it clearly. Next, run kubectl describe hpa <name>. ScalingActive=False means the metric is unreadable, often with FailedGetExternalMetric events. ScalingLimited=True means you hit maxReplicas or a behavior policy. Then load-test by pushing a synthetic backlog into a staging queue. Watch replicas rise, the backlog drain, and scale-in complete without redelivered messages.

In production, alert when the metric is stale. A frozen exporter makes the HPA stop scaling, and nothing visibly fails. Also check node capacity. Without a node autoscaler or spare headroom, new pods stay Pending while the backlog keeps growing.

Cost

With the adapter, you own the exporter, the Prometheus queries, and the adapter rules. KEDA puts most of that in one ScaledObject. Either way, compute savings come from matching replicas to load:

savings = (static replicas − average autoscaled replicas) × hours × cost per replica-hour

Illustrative estimate (assumed numbers, not a benchmark): 20 static replicas, 6 on average when autoscaled, 730 hours, $0.05 per replica-hour gives (20 − 6) × 730 × 0.05 = $511 per month.

Engineering time often matters more. Well-tuned scaling ends manual replica bumps and the backlog pages that come with them.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to hpa custom metrics queue depth, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.