Why is CPU the wrong signal for scaling queue workers?
A typical queue consumer spends most of its time waiting: on a broker fetch, a database write, an HTTP call to another service. A worker blocked on network I/O uses almost no CPU, so a pod can sit at a small fraction of its CPU request while thousands of messages pile up behind it. The default Horizontal Pod Autoscaler compares observed CPU or memory utilization against a target. If utilization stays low, the HPA has no reason to add replicas, and the backlog keeps growing.
The signal also points in the wrong direction. CPU describes how hard the current pods are working. It tells you nothing about how much work is still waiting. If a downstream dependency slows down, each worker does less CPU work per second, utilization drops, and a resource-based HPA may scale in exactly when throughput is falling and lag is rising. Memory is no better, since most consumers hold a roughly constant working set no matter how deep the queue gets.
Queue-based autoscaling bases the replica count on outstanding work instead: messages waiting in a queue, consumer group lag on a stream, or pending jobs in a list. That number grows when consumers fall behind and shrinks as they catch up, which is the behavior you want the autoscaler to follow. It also makes scale to zero possible. An empty queue is a clear sign that no workers are needed, whereas CPU utilization cannot be measured on a pod that doesn't exist.
This approach suits backend and platform engineers who run asynchronous Kubernetes workers: job processors, event consumers, and batch pipelines whose load comes in bursts. KEDA queue-based autoscaling gives the HPA custom metrics taken from the queue itself, so the HPA scales on the backlog instead of on resource usage.
How does KEDA work with the Kubernetes HPA?
KEDA does not replace the Horizontal Pod Autoscaler. It feeds the HPA and handles the cases the HPA cannot. It runs as two pieces:
- The operator watches
ScaledObjectresources. For each one, it creates and owns a standard HPA that targets your Deployment. It also polls the configured trigger directly, such as queue length or consumer lag. - The metrics adapter registers with the Kubernetes API server as a provider of the external metrics API. When the HPA controller asks for the metric, the adapter queries the broker and returns the value. From the HPA's side, this looks like any other external metric, and it applies its usual replica calculation.
The HPA cannot scale a workload to zero replicas in a standard cluster, and it cannot wake one up from zero. KEDA covers both gaps itself. When the backlog stays empty for the configured cooldown period, the operator sets the Deployment to zero replicas. When the backlog rises past the activation threshold, the operator scales the Deployment to one replica. From there, the HPA takes over scaling between one and the maximum.
The HPA that KEDA creates must be the only HPA on that Deployment. Two autoscalers targeting the same workload will overwrite each other's replica counts. If you already have a hand-written HPA, delete it when you apply the ScaledObject, in a single cutover. Do not run both in parallel.
The costs are operational. KEDA adds another control-plane component that you must upgrade and monitor. Autoscaling also now depends on two things that can fail:
- The broker's management API must be reachable.
- The credentials KEDA uses to read it must be valid.
If either breaks, the workers stop scaling.
+----------------------+ polls trigger (API + creds) +------------------+
| KEDA operator |----------------------------->| Broker mgmt API |
| watches ScaledObject | | (queue / lag) |
+----------------------+ +------------------+
| | ^
| creates | 0 -> 1 on activation threshold | queries
| and owns | 1 -> 0 after cooldown |
v | +------------------+
+--------------+ | external metric (via API server) | KEDA metrics |
| HPA |--+---------------------------------->| adapter |
| (only one on | | | (external API) |
| Deployment) | | +------------------+
+--------------+ |
| |
| 1 .. max |
v v
+---------------------------------+
| Worker Deployment |
+---------------------------------+
The KEDA operator creates and owns the only HPA on the Deployment and handles 0-to-1 and 1-to-0 itself. The HPA scales from 1 to max using external metrics that the KEDA adapter reads from the broker.
How do you configure a ScaledObject for a queue trigger?
A ScaledObject has two parts: the workload it scales and the triggers that drive it. In this example, the workload is a worker Deployment that consumes from Amazon SQS:
scaleTargetRefnames the Deployment (or StatefulSet) that KEDA hands to the HPA.minReplicaCount: 0enables scale to zero. Set it to 1 if cold-start latency matters more than idle cost.maxReplicaCountis a capacity limit for everything downstream, not for the cluster. If your database can handle 40 connections and each worker holds two, cap the Deployment at 20. A larger value only moves the backlog from the queue into your database.pollingInterval(default 30s) sets how often KEDA queries the broker. A shorter interval reacts faster. It also increases load on the broker's API, and on managed queues such as SQS each poll is a billable request.cooldownPeriod(default 300s) applies only to the final step from 1 replica to 0. Scaling between 1 and N is handled by the HPA's own behavior settings.
The trigger is aws-sqs-queue, with queueURL and queueLength as the per-replica target. Do not put credentials in the trigger metadata. Reference a TriggerAuthentication instead. That object either reads keys from a Kubernetes Secret or uses a pod identity provider, so KEDA assumes an IAM role and no static keys exist anywhere in the cluster.
Other brokers follow the same pattern with a different trigger type:
| Broker | Trigger type | Backlog field | Typical auth |
|---|---|---|---|
| RabbitMQ | rabbitmq |
queueName + value |
Secret with connection string |
| SQS | aws-sqs-queue |
queueLength |
Pod identity |
| Kafka | kafka |
lagThreshold (consumer lag) |
Secret with SASL/TLS material |
| Redis lists | redis |
listLength |
Secret with password |
# Credentials live in TriggerAuthentication, not in trigger metadata.
# With pod identity, KEDA assumes an IAM role and no static keys are stored.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: sqs-worker-auth
namespace: jobs
spec:
podIdentity:
provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sqs-worker
namespace: jobs
spec:
scaleTargetRef:
name: sqs-worker # Deployment that KEDA hands to the HPA
minReplicaCount: 0 # scale to zero when the queue is empty
maxReplicaCount: 20 # 40 DB connections / 2 per worker
pollingInterval: 30 # seconds; shorter means faster reaction and more billable SQS calls
cooldownPeriod: 300 # seconds before the final step from 1 to 0
triggers:
- type: aws-sqs-queue
authenticationRef:
name: sqs-worker-auth
metadata:
queueURL:
queueLength: "5" # target messages per replica
awsRegion: us-east-1
A TriggerAuthentication that uses AWS pod identity and a ScaledObject that scales an SQS worker Deployment between 0 and 20 replicas based on queue backlog.
How do you choose the per-replica target and activation threshold?
KEDA hands the HPA the queue length as an AverageValue metric. The HPA then computes desiredReplicas = ceil(totalQueueLength / target) and clamps the result between minReplicaCount and maxReplicaCount. The target is therefore the backlog you are willing to let each replica own. The best way to set it is from your latency budget rather than by trial and error.
Illustrative estimate. Assume each message takes 200 ms on a single-threaded worker, so one replica handles 5 messages per second. Assume you can tolerate a backlog draining in 60 seconds. Each replica can clear 5 × 60 = 300 messages in that window, so set the target to 300. A backlog of 6,000 then yields ceil(6000 / 300) = 20 replicas. If maxReplicaCount is lower than that, drain time stretches proportionally. If the workers share a database, the database may become the limit before the replica count does.
The activation threshold (for example activationQueueLength) is a separate setting. It only decides the 0↔1 transition, meaning whether KEDA treats the trigger as active at all. Above one replica, only the target matters. cooldownPeriod controls how long KEDA waits after the trigger was last active before scaling from one replica back to zero. It does not affect scaling between 1 and N, which the HPA governs.
Cold start from zero. The worst case is polling interval + image pull + pod startup. Illustrative estimate: 30 s polling + 20 s pull on a cold node + 10 s startup is roughly 60 s before the first message is processed. Pre-pulled images or minReplicaCount: 1 remove most of that delay.
Queue length measures backlog only. It tells you nothing about processing rate or how long messages have been waiting. Where the broker exposes it, scale on consumer lag (Kafka) or oldest-message age (for example SQS's ApproximateAgeOfOldestMessage).
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-worker
namespace: workers
spec:
scaleTargetRef:
name: order-worker # Deployment to scale
pollingInterval: 30 # seconds between queue checks; part of cold-start latency
cooldownPeriod: 300 # seconds after last active trigger before 1 -> 0
minReplicaCount: 0 # set to 1 to avoid the cold-start path entirely
maxReplicaCount: 20 # caps drain speed; check downstream DB capacity
triggers:
- type: rabbitmq
metadata:
queueName: orders
mode: QueueLength
# Per-replica target (illustrative estimate):
# 200 ms/msg -> 5 msg/s; 60 s drain budget -> 5 * 60 = 300
# HPA: desired = ceil(totalQueueLength / 300)
value: "300"
# Activation only gates 0 <-> 1; ignored once replicas >= 1
activationValue: "10"
authenticationRef:
name: rabbitmq-auth # TriggerAuthentication holding the AMQP host URI
A KEDA ScaledObject that sets a per-replica queue-length target derived from a latency budget, a separate activation threshold for scale from zero, and the polling and cooldown settings.
How do you stop replica counts from flapping?
A queue that fills in bursts and drains quickly produces a sawtooth metric. If the HPA follows it exactly, it removes pods as soon as the backlog clears and adds them back on the next burst. Each cycle pays for pod startup, connection setup, and cache warm-up again.
You control this through advanced.horizontalPodAutoscalerConfig.behavior in the ScaledObject. KEDA passes that block unchanged to the HPA it generates, so it uses the standard HPA behavior fields:
scaleDown.stabilizationWindowSeconds: the HPA uses the highest replica recommendation from this window. A window longer than the gap between your bursts keeps capacity in place through the quiet periods. The HPA default is 300 seconds.scaleDown.policies: these cap how fast replicas are removed, for example by a number ofPodsor aPercentperperiodSeconds. A gradual step-down protects you if the backlog returns halfway through a drain.scaleUppolicies: the scale-up window defaults to zero, which is usually correct for queues. Add aPodsorPercentlimit if a sudden spike would otherwise request dozens of replicas at once.
KEDA's cooldownPeriod is a separate setting. It only controls the final step from one replica to zero, not scaling between nonzero counts.
Fast scale-up has a cost. If you scale up aggressively and scale down slowly, pods can request more capacity than your nodes have. New replicas then stay Pending until the cluster autoscaler provisions nodes, which takes minutes rather than seconds. Size the scale-up limits to match how much spare capacity you keep in the cluster. Otherwise, tuning the HPA for faster scale-up only moves the delay to node provisioning.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-worker
spec:
scaleTargetRef:
name: order-worker # Deployment that consumes the queue
minReplicaCount: 0
maxReplicaCount: 40
cooldownPeriod: 300 # only governs the final 1 -> 0 step
advanced:
horizontalPodAutoscalerConfig:
behavior: # passed unchanged to the generated HPA
scaleDown:
# Longer than the typical gap between bursts, so pods survive quiet periods
stabilizationWindowSeconds: 600
policies:
- type: Percent # step down gradually in case the backlog returns
value: 25
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0 # react to new backlog immediately
selectPolicy: Min # apply the more conservative of the two limits
policies:
- type: Pods # cap per-minute growth to roughly the spare
value: 8 # node capacity, so new pods do not sit Pending
periodSeconds: 60
- type: Percent
value: 100
periodSeconds: 60
triggers:
- type: rabbitmq
metadata:
queueName: orders
mode: QueueLength
value: "50" # target messages per replica
authenticationRef:
name: rabbitmq-auth # TriggerAuthentication holding the connection string
A ScaledObject that holds capacity through bursty drains with a long scale-down window and gradual step-down, and caps scale-up growth to roughly the cluster's spare capacity so new replicas do not wait on the cluster autoscaler.
How do you shut down workers without losing messages?
Scale-down is the riskier direction. When the HPA removes a replica, Kubernetes sends the pod SIGTERM, waits up to terminationGracePeriodSeconds (30 seconds by default), and then sends SIGKILL. The HPA does not know which pod is mid-task, so any worker can be chosen.
On SIGTERM, a worker should stop fetching new messages, finish or abandon its in-flight work, and exit. Set the grace period above your worst-case processing time. If a task cannot finish in time, the worker should release it. Depending on the broker, that means negative-acknowledging it, letting its visibility timeout expire, or not committing its offset. The broker then redelivers it to another replica.
Acknowledge only after the work is durably complete. If you acknowledge on receipt and the pod is killed, the message is lost. If you acknowledge after completion, a kill causes redelivery instead. That makes duplicates unavoidable, so handlers must be idempotent: use deduplication keys, upserts, or conditional writes so that processing a message twice has the same effect as processing it once.
Some tasks take too long, or are too unsafe to interrupt, for any reasonable grace period. Examples are video encodes and multi-hour imports. For these, use a ScaledJob. KEDA creates one Kubernetes Job per message or batch, and a falling queue length does not terminate Jobs that are already running.
| ScaledObject | ScaledJob | |
|---|---|---|
| Workload | Long-lived Deployment pods | One Job per message or batch |
| Scale-down | HPA terminates running pods | Running Jobs complete |
| Best for | Short, interruptible tasks | Long or non-interruptible tasks |
| Startup cost | Paid once per pod | Paid on every Job |
How do you verify and debug KEDA autoscaling?
Start with the HPA that KEDA generates, not the ScaledObject. KEDA hands the actual scaling decision to the HPA, so that is where you see what is happening. kubectl get hpa lists the object, usually named keda-hpa-<scaledobject-name>, with current versus target metric values and the replica count. kubectl describe hpa shows the conditions and events. Look at these three conditions:
ScalingActive=Falsemeans the HPA cannot read the metric.AbleToScale=Falsepoints to a problem with the scale target.ScalingLimited=Truemeans you hitmaxReplicaCount, not a metric limit.
Next, confirm the value the HPA actually receives. Query the external metrics API with kubectl get --raw against /apis/external.metrics.k8s.io/v1beta1/namespaces/<ns>/<metric-name>. Compare the result with the queue depth your broker reports. If the two disagree, the fault is in the scaler configuration, such as the wrong queue name, the wrong host, or the wrong unit, and not in the HPA.
kubectl get scaledobject shows READY and ACTIVE at a glance. The KEDA operator logs and the metrics API server logs explain why either one is false. Authentication failures and unreachable brokers appear there first.
Then load-test the setup. Push a known number of messages, watch kubectl get hpa -w, and check three things:
- the replicas scale up within the polling interval plus pod startup time
- the replicas settle at roughly backlog divided by target
- the replicas return to the minimum after the cooldown ends
Finally, alert on KEDA itself. If broker credentials expire or the operator crashes, the HPA keeps its last replica count and nothing pages you. Scrape KEDA's Prometheus metrics and alert on these conditions:
- scaler errors
- a
READY=FalseScaledObject - operator restarts
- a queue that is growing while the replica count stays flat
When should you avoid KEDA, and what are the alternatives?
Skip KEDA when the first message after an idle period cannot wait for a cold start. With scale to zero, that message waits for scheduling, an image pull and application startup. For latency-sensitive paths, set minReplicaCount to at least 1, or drop the queue trigger.
Also skip it when a downstream system is the real bottleneck. If workers write to a database that is already saturated, adding replicas increases contention and retries without draining the queue any faster. In that case, cap concurrency to match what the downstream system can handle.
Finally, KEDA is another operator and another external metrics API server that someone must upgrade, monitor and secure. A team that will not own that should not adopt it.
| Approach | Setup effort | Ongoing development cost | Operational overhead |
|---|---|---|---|
| KEDA | Moderate: install the operator, then write a ScaledObject per workload | Low: change a trigger value | Moderate: the operator, its metrics and its credentials |
| Resource-based HPA | Low | Low, but CPU tracks backlog poorly | Low: uses the built-in metrics pipeline |
| HPA with a separate metrics adapter | High: write metric queries and adapter rules | Moderate: adapter rules change whenever metrics change | Moderate: the adapter, plus a metrics backend |
| Fixed replicas | Minimal | Periodic manual resizing | Low, but you pay for idle capacity |
| Scheduled scaling | Low to moderate | Schedules drift away from real traffic | Low |
Declarative queue-based scaling removes the work that manual capacity tuning or a custom scaling script would otherwise require. A homegrown script that polls the queue and calls kubectl scale needs:
- RBAC permissions
- retry logic
- protection against two instances acting at once
- its own monitoring
Someone also has to maintain the script indefinitely. A ScaledObject replaces all of that with a reviewed manifest that sits in Git next to the Deployment it scales.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to keda queue based autoscaling, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.