Why is CPU the wrong signal for scaling queue workers?

A typical queue consumer spends most of its time waiting: on a broker fetch, a database write, an HTTP call to another service. A worker blocked on network I/O uses almost no CPU, so a pod can sit at a small fraction of its CPU request while thousands of messages pile up behind it. The default Horizontal Pod Autoscaler compares observed CPU or memory utilization against a target. If utilization stays low, the HPA has no reason to add replicas, and the backlog keeps growing.

The signal also points in the wrong direction. CPU describes how hard the current pods are working. It tells you nothing about how much work is still waiting. If a downstream dependency slows down, each worker does less CPU work per second, utilization drops, and a resource-based HPA may scale in exactly when throughput is falling and lag is rising. Memory is no better, since most consumers hold a roughly constant working set no matter how deep the queue gets.

Queue-based autoscaling bases the replica count on outstanding work instead: messages waiting in a queue, consumer group lag on a stream, or pending jobs in a list. That number grows when consumers fall behind and shrinks as they catch up, which is the behavior you want the autoscaler to follow. It also makes scale to zero possible. An empty queue is a clear sign that no workers are needed, whereas CPU utilization cannot be measured on a pod that doesn't exist.

This approach suits backend and platform engineers who run asynchronous Kubernetes workers: job processors, event consumers, and batch pipelines whose load comes in bursts. KEDA queue-based autoscaling gives the HPA custom metrics taken from the queue itself, so the HPA scales on the backlog instead of on resource usage.

How does KEDA work with the Kubernetes HPA?

KEDA does not replace the Horizontal Pod Autoscaler. It feeds the HPA and handles the cases the HPA cannot. It runs as two pieces:

  • The operator watches ScaledObject resources. For each one, it creates and owns a standard HPA that targets your Deployment. It also polls the configured trigger directly, such as queue length or consumer lag.
  • The metrics adapter registers with the Kubernetes API server as a provider of the external metrics API. When the HPA controller asks for the metric, the adapter queries the broker and returns the value. From the HPA's side, this looks like any other external metric, and it applies its usual replica calculation.

The HPA cannot scale a workload to zero replicas in a standard cluster, and it cannot wake one up from zero. KEDA covers both gaps itself. When the backlog stays empty for the configured cooldown period, the operator sets the Deployment to zero replicas. When the backlog rises past the activation threshold, the operator scales the Deployment to one replica. From there, the HPA takes over scaling between one and the maximum.

The HPA that KEDA creates must be the only HPA on that Deployment. Two autoscalers targeting the same workload will overwrite each other's replica counts. If you already have a hand-written HPA, delete it when you apply the ScaledObject, in a single cutover. Do not run both in parallel.

The costs are operational. KEDA adds another control-plane component that you must upgrade and monitor. Autoscaling also now depends on two things that can fail:

  • The broker's management API must be reachable.
  • The credentials KEDA uses to read it must be valid.

If either breaks, the workers stop scaling.

+----------------------+  polls trigger (API + creds) +------------------+
|    KEDA operator     |----------------------------->|  Broker mgmt API |
| watches ScaledObject |                              | (queue / lag)    |
+----------------------+                              +------------------+
   |              |                                            ^
   | creates      | 0 -> 1 on activation threshold             | queries
   | and owns     | 1 -> 0 after cooldown                      |
   v              |                                    +------------------+
+--------------+  | external metric (via API server)  |  KEDA metrics    |
|     HPA      |--+---------------------------------->|  adapter         |
| (only one on |  |                                   | (external API)   |
|  Deployment) |  |                                   +------------------+
+--------------+  |
   |              |
   | 1 .. max     |
   v              v
+---------------------------------+
|        Worker Deployment        |
+---------------------------------+

The KEDA operator creates and owns the only HPA on the Deployment and handles 0-to-1 and 1-to-0 itself. The HPA scales from 1 to max using external metrics that the KEDA adapter reads from the broker.

How do you configure a ScaledObject for a queue trigger?

A ScaledObject has two parts: the workload it scales and the triggers that drive it. In this example, the workload is a worker Deployment that consumes from Amazon SQS:

  • scaleTargetRef names the Deployment (or StatefulSet) that KEDA hands to the HPA.
  • minReplicaCount: 0 enables scale to zero. Set it to 1 if cold-start latency matters more than idle cost.
  • maxReplicaCount is a capacity limit for everything downstream, not for the cluster. If your database can handle 40 connections and each worker holds two, cap the Deployment at 20. A larger value only moves the backlog from the queue into your database.
  • pollingInterval (default 30s) sets how often KEDA queries the broker. A shorter interval reacts faster. It also increases load on the broker's API, and on managed queues such as SQS each poll is a billable request.
  • cooldownPeriod (default 300s) applies only to the final step from 1 replica to 0. Scaling between 1 and N is handled by the HPA's own behavior settings.

The trigger is aws-sqs-queue, with queueURL and queueLength as the per-replica target. Do not put credentials in the trigger metadata. Reference a TriggerAuthentication instead. That object either reads keys from a Kubernetes Secret or uses a pod identity provider, so KEDA assumes an IAM role and no static keys exist anywhere in the cluster.

Other brokers follow the same pattern with a different trigger type:

Broker Trigger type Backlog field Typical auth
RabbitMQ rabbitmq queueName + value Secret with connection string
SQS aws-sqs-queue queueLength Pod identity
Kafka kafka lagThreshold (consumer lag) Secret with SASL/TLS material
Redis lists redis listLength Secret with password
# Credentials live in TriggerAuthentication, not in trigger metadata.
# With pod identity, KEDA assumes an IAM role and no static keys are stored.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: sqs-worker-auth
  namespace: jobs
spec:
  podIdentity:
    provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-worker
  namespace: jobs
spec:
  scaleTargetRef:
    name: sqs-worker        # Deployment that KEDA hands to the HPA
  minReplicaCount: 0        # scale to zero when the queue is empty
  maxReplicaCount: 20       # 40 DB connections / 2 per worker
  pollingInterval: 30       # seconds; shorter means faster reaction and more billable SQS calls
  cooldownPeriod: 300       # seconds before the final step from 1 to 0
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: sqs-worker-auth
      metadata:
        queueURL:
        queueLength: "5"    # target messages per replica
        awsRegion: us-east-1

A TriggerAuthentication that uses AWS pod identity and a ScaledObject that scales an SQS worker Deployment between 0 and 20 replicas based on queue backlog.

How do you choose the per-replica target and activation threshold?

KEDA hands the HPA the queue length as an AverageValue metric. The HPA then computes desiredReplicas = ceil(totalQueueLength / target) and clamps the result between minReplicaCount and maxReplicaCount. The target is therefore the backlog you are willing to let each replica own. The best way to set it is from your latency budget rather than by trial and error.

Illustrative estimate. Assume each message takes 200 ms on a single-threaded worker, so one replica handles 5 messages per second. Assume you can tolerate a backlog draining in 60 seconds. Each replica can clear 5 × 60 = 300 messages in that window, so set the target to 300. A backlog of 6,000 then yields ceil(6000 / 300) = 20 replicas. If maxReplicaCount is lower than that, drain time stretches proportionally. If the workers share a database, the database may become the limit before the replica count does.

The activation threshold (for example activationQueueLength) is a separate setting. It only decides the 0↔1 transition, meaning whether KEDA treats the trigger as active at all. Above one replica, only the target matters. cooldownPeriod controls how long KEDA waits after the trigger was last active before scaling from one replica back to zero. It does not affect scaling between 1 and N, which the HPA governs.

Cold start from zero. The worst case is polling interval + image pull + pod startup. Illustrative estimate: 30 s polling + 20 s pull on a cold node + 10 s startup is roughly 60 s before the first message is processed. Pre-pulled images or minReplicaCount: 1 remove most of that delay.

Queue length measures backlog only. It tells you nothing about processing rate or how long messages have been waiting. Where the broker exposes it, scale on consumer lag (Kafka) or oldest-message age (for example SQS's ApproximateAgeOfOldestMessage).

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
  namespace: workers
spec:
  scaleTargetRef:
    name: order-worker          # Deployment to scale
  pollingInterval: 30           # seconds between queue checks; part of cold-start latency
  cooldownPeriod: 300           # seconds after last active trigger before 1 -> 0
  minReplicaCount: 0            # set to 1 to avoid the cold-start path entirely
  maxReplicaCount: 20           # caps drain speed; check downstream DB capacity
  triggers:
    - type: rabbitmq
      metadata:
        queueName: orders
        mode: QueueLength
        # Per-replica target (illustrative estimate):
        # 200 ms/msg -> 5 msg/s; 60 s drain budget -> 5 * 60 = 300
        # HPA: desired = ceil(totalQueueLength / 300)
        value: "300"
        # Activation only gates 0 <-> 1; ignored once replicas >= 1
        activationValue: "10"
      authenticationRef:
        name: rabbitmq-auth     # TriggerAuthentication holding the AMQP host URI

A KEDA ScaledObject that sets a per-replica queue-length target derived from a latency budget, a separate activation threshold for scale from zero, and the polling and cooldown settings.

How do you stop replica counts from flapping?

A queue that fills in bursts and drains quickly produces a sawtooth metric. If the HPA follows it exactly, it removes pods as soon as the backlog clears and adds them back on the next burst. Each cycle pays for pod startup, connection setup, and cache warm-up again.

You control this through advanced.horizontalPodAutoscalerConfig.behavior in the ScaledObject. KEDA passes that block unchanged to the HPA it generates, so it uses the standard HPA behavior fields:

  • scaleDown.stabilizationWindowSeconds: the HPA uses the highest replica recommendation from this window. A window longer than the gap between your bursts keeps capacity in place through the quiet periods. The HPA default is 300 seconds.
  • scaleDown.policies: these cap how fast replicas are removed, for example by a number of Pods or a Percent per periodSeconds. A gradual step-down protects you if the backlog returns halfway through a drain.
  • scaleUp policies: the scale-up window defaults to zero, which is usually correct for queues. Add a Pods or Percent limit if a sudden spike would otherwise request dozens of replicas at once.

KEDA's cooldownPeriod is a separate setting. It only controls the final step from one replica to zero, not scaling between nonzero counts.

Fast scale-up has a cost. If you scale up aggressively and scale down slowly, pods can request more capacity than your nodes have. New replicas then stay Pending until the cluster autoscaler provisions nodes, which takes minutes rather than seconds. Size the scale-up limits to match how much spare capacity you keep in the cluster. Otherwise, tuning the HPA for faster scale-up only moves the delay to node provisioning.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker            # Deployment that consumes the queue
  minReplicaCount: 0
  maxReplicaCount: 40
  cooldownPeriod: 300             # only governs the final 1 -> 0 step
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:                   # passed unchanged to the generated HPA
        scaleDown:
          # Longer than the typical gap between bursts, so pods survive quiet periods
          stabilizationWindowSeconds: 600
          policies:
            - type: Percent       # step down gradually in case the backlog returns
              value: 25
              periodSeconds: 60
        scaleUp:
          stabilizationWindowSeconds: 0   # react to new backlog immediately
          selectPolicy: Min       # apply the more conservative of the two limits
          policies:
            - type: Pods          # cap per-minute growth to roughly the spare
              value: 8            # node capacity, so new pods do not sit Pending
              periodSeconds: 60
            - type: Percent
              value: 100
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      metadata:
        queueName: orders
        mode: QueueLength
        value: "50"               # target messages per replica
      authenticationRef:
        name: rabbitmq-auth       # TriggerAuthentication holding the connection string

A ScaledObject that holds capacity through bursty drains with a long scale-down window and gradual step-down, and caps scale-up growth to roughly the cluster's spare capacity so new replicas do not wait on the cluster autoscaler.

How do you shut down workers without losing messages?

Scale-down is the riskier direction. When the HPA removes a replica, Kubernetes sends the pod SIGTERM, waits up to terminationGracePeriodSeconds (30 seconds by default), and then sends SIGKILL. The HPA does not know which pod is mid-task, so any worker can be chosen.

On SIGTERM, a worker should stop fetching new messages, finish or abandon its in-flight work, and exit. Set the grace period above your worst-case processing time. If a task cannot finish in time, the worker should release it. Depending on the broker, that means negative-acknowledging it, letting its visibility timeout expire, or not committing its offset. The broker then redelivers it to another replica.

Acknowledge only after the work is durably complete. If you acknowledge on receipt and the pod is killed, the message is lost. If you acknowledge after completion, a kill causes redelivery instead. That makes duplicates unavoidable, so handlers must be idempotent: use deduplication keys, upserts, or conditional writes so that processing a message twice has the same effect as processing it once.

Some tasks take too long, or are too unsafe to interrupt, for any reasonable grace period. Examples are video encodes and multi-hour imports. For these, use a ScaledJob. KEDA creates one Kubernetes Job per message or batch, and a falling queue length does not terminate Jobs that are already running.

ScaledObject ScaledJob
Workload Long-lived Deployment pods One Job per message or batch
Scale-down HPA terminates running pods Running Jobs complete
Best for Short, interruptible tasks Long or non-interruptible tasks
Startup cost Paid once per pod Paid on every Job

How do you verify and debug KEDA autoscaling?

Start with the HPA that KEDA generates, not the ScaledObject. KEDA hands the actual scaling decision to the HPA, so that is where you see what is happening. kubectl get hpa lists the object, usually named keda-hpa-<scaledobject-name>, with current versus target metric values and the replica count. kubectl describe hpa shows the conditions and events. Look at these three conditions:

  • ScalingActive=False means the HPA cannot read the metric.
  • AbleToScale=False points to a problem with the scale target.
  • ScalingLimited=True means you hit maxReplicaCount, not a metric limit.

Next, confirm the value the HPA actually receives. Query the external metrics API with kubectl get --raw against /apis/external.metrics.k8s.io/v1beta1/namespaces/<ns>/<metric-name>. Compare the result with the queue depth your broker reports. If the two disagree, the fault is in the scaler configuration, such as the wrong queue name, the wrong host, or the wrong unit, and not in the HPA.

kubectl get scaledobject shows READY and ACTIVE at a glance. The KEDA operator logs and the metrics API server logs explain why either one is false. Authentication failures and unreachable brokers appear there first.

Then load-test the setup. Push a known number of messages, watch kubectl get hpa -w, and check three things:

  • the replicas scale up within the polling interval plus pod startup time
  • the replicas settle at roughly backlog divided by target
  • the replicas return to the minimum after the cooldown ends

Finally, alert on KEDA itself. If broker credentials expire or the operator crashes, the HPA keeps its last replica count and nothing pages you. Scrape KEDA's Prometheus metrics and alert on these conditions:

  • scaler errors
  • a READY=False ScaledObject
  • operator restarts
  • a queue that is growing while the replica count stays flat

When should you avoid KEDA, and what are the alternatives?

Skip KEDA when the first message after an idle period cannot wait for a cold start. With scale to zero, that message waits for scheduling, an image pull and application startup. For latency-sensitive paths, set minReplicaCount to at least 1, or drop the queue trigger.

Also skip it when a downstream system is the real bottleneck. If workers write to a database that is already saturated, adding replicas increases contention and retries without draining the queue any faster. In that case, cap concurrency to match what the downstream system can handle.

Finally, KEDA is another operator and another external metrics API server that someone must upgrade, monitor and secure. A team that will not own that should not adopt it.

Approach Setup effort Ongoing development cost Operational overhead
KEDA Moderate: install the operator, then write a ScaledObject per workload Low: change a trigger value Moderate: the operator, its metrics and its credentials
Resource-based HPA Low Low, but CPU tracks backlog poorly Low: uses the built-in metrics pipeline
HPA with a separate metrics adapter High: write metric queries and adapter rules Moderate: adapter rules change whenever metrics change Moderate: the adapter, plus a metrics backend
Fixed replicas Minimal Periodic manual resizing Low, but you pay for idle capacity
Scheduled scaling Low to moderate Schedules drift away from real traffic Low

Declarative queue-based scaling removes the work that manual capacity tuning or a custom scaling script would otherwise require. A homegrown script that polls the queue and calls kubectl scale needs:

  • RBAC permissions
  • retry logic
  • protection against two instances acting at once
  • its own monitoring

Someone also has to maintain the script indefinitely. A ScaledObject replaces all of that with a reviewed manifest that sits in Git next to the Deployment it scales.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to keda queue based autoscaling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.