KEDA Queue Depth Autoscaling: Scaling APIs and Workers on Backlog

Why CPU is the wrong signal for queue-fed APIs and workers

A worker that pulls a message, calls a database or a downstream API, and waits for the response spends most of its time blocked on I/O. Its CPU and memory stay flat while the backlog grows. A CPU-based Horizontal Pod Autoscaler (HPA) sees an idle pod and does nothing, or scales only after the backlog has already turned into user-visible latency. CPU measures how hard pods are working, not how much work is waiting.

Queue depth is the number of messages waiting to be consumed. Consumer lag, used by log-based systems, is how far a consumer group is behind the newest offset. Message age is how long the oldest unprocessed message has been waiting. All three measure pending work directly, so the autoscaler can act on the backlog itself. Queue-depth autoscaling sets the replica count from this backlog divided by the throughput each replica can sustain.

This approach suits backend and platform engineers who run async workers, queue-fed APIs, or event-driven autoscaling pipelines, where demand arrives in bursts and idle periods are common.

How KEDA fits into the Kubernetes autoscaling stack

KEDA does not replace the Horizontal Pod Autoscaler. It extends it. KEDA runs as two components: an operator that watches ScaledObject resources, and a metrics server that registers with the Kubernetes external metrics API. When you create a ScaledObject for a Deployment, the operator creates and manages an HPA for that Deployment. You configure the ScaledObject and leave the generated HPA alone, because KEDA reconciles it.

The metric path for a queue-fed workload looks like this:

1. Broker: the queue holds the backlog.
2. KEDA metrics server: the scaler queries the broker for queue length.
3. External metrics API: the HPA controller reads the value as an external metric.
4. HPA: compares the metric to the per-replica target and computes desired replicas.
5. Deployment: the replica count changes, and the scheduler places the new pods.

The work is split for a reason. The HPA handles scaling between 1 and N replicas well, but it cannot activate a workload from zero, because there are no pods and no baseline to scale from. The KEDA operator covers that case. It polls the trigger on its own interval and, when the activation threshold is crossed, sets the Deployment to one replica. From there, the HPA takes over.

Pods are only half of the problem. If the nodes are full, the new replicas stay Pending until the cluster autoscaler or your node provisioner adds capacity. That delay adds directly to your scale-out latency.

KEDA is now on the scaling critical path. If its metrics server is unavailable, the HPA cannot read the metric and stops making scaling decisions. Treat KEDA as production infrastructure: run it with redundancy, monitor it, and alert on it.

Copy+------------------+                                +----------------------+
|  Message broker  |<-- query queue length ---------|  KEDA metrics server |
|  (queue backlog) |                                |  (scaler)            |
+------------------+                                +----------------------+
        ^                                                      |
        | poll trigger (activation)                            | serves metric
        |                                                      v
+------------------+                                +----------------------+
|  KEDA operator   |                                | External metrics API |
|  watches         |                                +----------------------+
|  ScaledObject    |                                           | read metric
+------------------+                                           v
   |        |                                       +----------------------+
   | 0 -> 1 +-- creates, reconciles --------------->|  HPA (1 -> N)        |
   |                                                +----------------------+
   |                                                           | desired replicas
   v                                                           v
+--------------------------------------------------------------------------+
|  Deployment: replica count changes, scheduler places pods                |
+--------------------------------------------------------------------------+
        |
        | new pods Pending if nodes are full (adds scale-out latency)
        v
+--------------------------------------------------------------------------+
|  Cluster autoscaler / node provisioner adds capacity                     |
+--------------------------------------------------------------------------+

KEDA's metrics server serves queue length to the HPA through the external metrics API for 1-to-N scaling, while the KEDA operator handles 0-to-1 activation, which puts KEDA on the scaling critical path above node provisioning.

Sizing the per-replica target from measured throughput

For an average-value target like queue length, the HPA computes desired replicas as roughly the current metric divided by the per-replica target, rounded up. With 4,200 messages queued and a target of 500, it asks for 9 replicas. The target is the only number you control, so it determines how much backlog each pod is expected to clear.

Illustrative estimate: assume each pod processes 50 messages/second and you accept about 10 seconds of queueing delay. Then 50 × 10 = 500 messages per replica. At that target, a pod's share of the backlog drains in about 10 seconds.

Measuring your own numbers

Measure throughput under realistic load, not in a microbenchmark. Run one replica against a pre-filled queue with production-like messages and real downstream dependencies. Record sustained messages/second after warm-up. Use a conservative figure rather than the peak, because the HPA assumes every replica performs at that rate.

Choose the latency budget from what callers or downstream SLOs can tolerate. Then subtract time for pod scheduling and startup, since new replicas contribute nothing until they are ready.

Re-measure after code changes, dependency upgrades, or shifts in message mix. A slower database query silently turns a 10-second budget into 30.

When queue length misleads

Queue length counts messages, not work. A thousand cheap messages and a thousand expensive ones look identical to the HPA. If cost varies widely, you have two main options:

• Scale on consumer lag or the age of the oldest message, which reflects actual delay.
• Split queues by work type so each queue has a homogeneous target.

Implementing a ScaledObject with a queue-length trigger

Start with scaleTargetRef, which points at the Deployment that consumes the queue. KEDA creates and owns the HPA for that Deployment, so delete any HPA you defined by hand. Set minReplicaCount to the floor you need for latency, and set maxReplicaCount from the capacity of your downstream dependencies, not from the queue. A backlog of a million messages should not open a thousand database connections.

pollingInterval controls how often KEDA queries the broker. cooldownPeriod applies only when KEDA scales between zero and one replica. Scaling between one and N replicas is handled by the HPA's own behavior settings.

In the trigger, use queue-length mode and set the target value to the per-replica figure you derived in the previous section. Do not put the connection string inline. Instead, set authenticationRef to a TriggerAuthentication that maps trigger parameters to keys in a Kubernetes Secret through secretTargetRef. This lets you rotate credentials without editing the ScaledObject.

The default HPA behavior is a poor fit for bursty queues. Scale-up has no stabilization window by default. Scale-down waits a fixed window and can then remove many pods at once. Under advanced.horizontalPodAutoscalerConfig.behavior, configure each direction separately:

• Scale-up: allow fast growth, for example a percentage policy over a short period, with a small stabilization window.
• Scale-down: use a longer stabilization window and a policy that removes a few pods per period, so a briefly drained queue does not cause flapping.

If you add CPU or memory triggers alongside the queue trigger, be aware of how the HPA combines them. It computes a replica count for each metric and uses the highest one. That makes it hard to tell which signal caused a given scaling event, so log and graph each metric's proposal separately.

CopyapiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbitmq-auth
spec:
  secretTargetRef:
    - parameter: host        # AMQP URI comes from the Secret, never from this manifest
      name: rabbitmq-conn    # existing Kubernetes Secret (placeholder name)
      key: host
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker       # Deployment that consumes the queue; KEDA owns its HPA
  minReplicaCount: 2         # latency floor
  maxReplicaCount: 20        # capped by downstream capacity (e.g. DB connections), not queue size
  pollingInterval: 15        # seconds between broker queries
  cooldownPeriod: 300        # only used for 1 -> 0; irrelevant while minReplicaCount > 0
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 30   # small window: react fast, ignore single-poll spikes
          policies:
            - {type: Percent, value: 100, periodSeconds: 30}  # at most double every 30s
        scaleDown:
          stabilizationWindowSeconds: 300  # wait 5 min before trusting a drained queue
          policies:
            - {type: Pods, value: 2, periodSeconds: 60}       # remove at most 2 pods per minute
  triggers:
    - type: rabbitmq
      authenticationRef:
        name: rabbitmq-auth
      metadata:
        queueName: orders
        mode: QueueLength
        value: "50"          # per-replica backlog target derived in the previous section

A RabbitMQ queue-length ScaledObject with Secret-backed TriggerAuthentication and separate scale-up and scale-down HPA behavior to avoid flapping on bursty queues.

Scale to zero: when to use it and when to avoid it

Scale to zero works when nobody is waiting on the response. Batch jobs, async workers and background consumers can tolerate a delay before the first message is processed. For them, an idle deployment that costs nothing is a good trade.

KEDA handles the 0→1 transition itself. The HPA only manages scaling between 1 and N replicas. The activation threshold decides whether the workload should exist at all. The scaling target decides how many replicas run once it does. Set activation above zero if a single stray message should not wake the deployment.

Cold start is the sum of four delays:

• Polling interval: KEDA only sees the backlog on its next poll.
• Pod scheduling: the pod may wait for a node, and possibly for the cluster autoscaler to add one.
• Image pull: large images on nodes without a cache add the most variance.
• App startup: the process must boot, warm caches and connect before it passes readiness checks.

A synchronous API cannot absorb that sum on a user request, so keep minReplicaCount at 1 or higher.

Queue-depth scaling also has limits beyond scale to zero. Kafka assigns each partition to at most one consumer in a group, so replicas beyond the partition count sit idle. If a downstream database or rate-limited API cannot absorb more parallel consumers, adding replicas only moves the backlog downstream.

Making scale-out and scale-in safe for the rest of the platform

Set maxReplicaCount from the tightest downstream limit, not from backlog size. Suppose the database allows 200 connections, other services already hold 80, and each pod opens a pool of 10. That gives a ceiling of 12 replicas (illustrative estimate). Apply the same arithmetic to broker connection quotas and third-party API rate limits, and use whichever ceiling is lowest.

Capacity and disruption

Accurate resource requests matter for two reasons. The scheduler packs nodes using requests, and a cluster autoscaler or node provisioner only adds nodes after pods go Pending. A sudden backlog can therefore wait minutes for new capacity. If that lag is unacceptable, raise minReplicaCount or keep spare node headroom. A PodDisruptionBudget limits voluntary evictions during node drains, but it does not restrict HPA scale-down.

Graceful shutdown

On SIGTERM, a consumer should do three things:

1. Stop fetching new messages.
2. Finish and ack the messages already in flight.
3. Nack or requeue anything it cannot complete before the deadline.

Set terminationGracePeriodSeconds above your worst-case processing time plus a margin. If the pod is still running when the grace period ends, the kubelet sends SIGKILL, and any unacked messages are redelivered as duplicates. Handlers must be idempotent regardless. Pods that also serve HTTP should fail readiness before draining.

KEDA's own availability

Once KEDA is in the scaling path, its failures become your failures. Run more than one operator replica, alert on its errors and on ScaledObject readiness, and configure a fallback replica count so the workload has a defined size when metrics are unavailable. Expired broker credentials are the typical silent failure: metric queries fail and scaling freezes at the current replica count. Rotate credentials through TriggerAuthentication and alert before they expire.

Copyimport signal, threading, time

GRACE_SECONDS = 120   # keep equal to terminationGracePeriodSeconds in the pod spec
SAFETY_MARGIN = 15    # time reserved to nack leftovers before the kubelet sends SIGKILL

stopping = threading.Event()
stop_deadline = None

def on_sigterm(signum, frame):
    # SIGTERM starts the grace period: stop intake and record the hard deadline.
    global stop_deadline
    stop_deadline = time.monotonic() + GRACE_SECONDS - SAFETY_MARGIN
    stopping.set()

signal.signal(signal.SIGTERM, on_sigterm)

def run(queue, handle):
    # queue is a thin wrapper over your broker client exposing fetch/ack/nack/close.
    while not stopping.is_set():
        batch = queue.fetch(max_messages=10, timeout=5)  # 1. intake stops once stopping is set
        for i, msg in enumerate(batch):
            if stopping.is_set() and time.monotonic() > stop_deadline:
                # 3. Out of time: hand unprocessed messages back for another consumer.
                for leftover in batch[i:]:
                    queue.nack(leftover, requeue=True)
                break
            handle(msg)    # must be idempotent; redelivery can still happen after SIGKILL
            queue.ack(msg) # 2. ack only after the work is durably done
    queue.close()

# Matching pod spec fields (Deployment .spec.template.spec):
#   terminationGracePeriodSeconds: 120   # > worst-case batch time + margin
#   containers[].resources.requests      # accurate CPU/memory so scheduling and node scale-up are predictable
#   containers[].readinessProbe          # HTTP-serving pods should fail this before draining

A queue consumer that stops fetching on SIGTERM, finishes and acks in-flight work, and requeues anything left before a deadline derived from terminationGracePeriodSeconds, with the matching pod spec fields noted at the end.

Verifying the setup and weighing cost and engineering time

Verify the wiring

KEDA creates a regular HPA for each ScaledObject, usually named with a keda-hpa- prefix. Run kubectl get hpa to confirm it exists and that the TARGETS column shows a real value instead of <unknown>. <unknown> means the metric query is failing. Next, run kubectl describe hpa on it to read the events and see the external metric the HPA is consuming. Then run kubectl get scaledobject and kubectl describe scaledobject to check the Ready and Active conditions. Ready tells you KEDA can reach the broker. Active tells you the trigger currently sees work.

Finally, run a controlled load test. Publish a fixed burst of messages to a staging queue, then stop. Watch replica count and backlog together over time. You should see replicas rise within one polling interval plus pod startup time, the backlog drain at roughly the per-replica rate you measured, and replicas step down only after the cooldown and stabilization windows expire. If the backlog drains but replicas stay high, revisit scale-in behavior.

Cost and engineering time

The savings come from two places. You run fewer idle replicas between peaks, and intermittent workers can scale to zero. Against that, you pay for:

• Operating KEDA itself
• The extra polling queries KEDA sends to the broker
• The upfront throughput measurement
• Retuning whenever handler latency changes

The larger gain is usually engineering time. Fewer manual replica bumps are needed during backlog incidents, and declarative triggers replace a hand-rolled custom-metrics adapter that someone would otherwise have to maintain.

Copy#!/usr/bin/env bash
# Verify a KEDA ScaledObject end to end in staging. Names are placeholders.
set -euo pipefail
NS=staging
SO=orders-worker            # ScaledObject name
DEPLOY=orders-worker        # Deployment it scales
HPA=keda-hpa-${SO}          # KEDA's usual naming for the generated HPA

# 1. The generated HPA exists, and TARGETS shows a value rather than <unknown>.
kubectl get hpa "$HPA" -n "$NS"
# Events and the external metric the HPA consumes.
kubectl describe hpa "$HPA" -n "$NS"

# 2. ScaledObject conditions: Ready means the broker is reachable,
#    Active means the trigger currently sees work.
kubectl get scaledobject "$SO" -n "$NS"
kubectl get scaledobject "$SO" -n "$NS" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

# 3. Controlled load test. Publish a fixed burst to the staging queue, then stop.
#    Replace this with your broker's own publish tool (placeholder command).
# ./publish-test-messages --queue orders-staging --count 5000

# 4. Watch replicas and the HPA's view of the backlog together over time.
#    Expect a rise within one polling interval plus pod startup, a drain near
#    the measured per-replica rate, and a step-down only after cooldown.
while true; do
  replicas=$(kubectl get deploy "$DEPLOY" -n "$NS" -o jsonpath='{.status.replicas}')
  targets=$(kubectl get hpa "$HPA" -n "$NS" --no-headers | awk '{print $3}')
  printf '%s replicas=%s hpa_targets=%s\n' "$(date +%T)" "${replicas:-0}" "$targets"
  sleep 15
done

A staging verification script that inspects the KEDA-generated HPA and the ScaledObject's Ready/Active conditions, then logs replica count next to the HPA's metric value while a fixed burst of messages drains.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to keda queue depth autoscaling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.