Why is CPU the wrong signal for scaling queue consumers?
A typical RabbitMQ consumer spends most of each message waiting. It waits on a database write, an HTTP call to another service, or an object storage upload. While it waits, the process uses almost no CPU. If you scale that consumer with a standard Horizontal Pod Autoscaler (HPA) targeting 70% CPU, the autoscaler sees idle pods. Meanwhile the queue grows from hundreds of messages to hundreds of thousands. Nothing scales, because by the metric you chose, nothing is wrong.
The underlying problem is that CPU and memory are symptoms of load, not the load itself. For a CPU-bound service handling synchronous requests, the symptom tracks the cause closely enough. For an I/O-bound worker it does not. Adding replicas does help the backlog drain faster, but the HPA has no signal telling it to add them.
Queue depth autoscaling uses the number of messages waiting in the broker as the scaling metric. Messages waiting is the quantity you actually care about, and it is visible before any downstream symptom appears. It also works in the other direction: when the queue is empty, you know the extra replicas are doing nothing.
This approach fits teams running asynchronous workers on RabbitMQ or a similar broker when:
- Work arrives in bursts, such as nightly imports, webhook storms or batch jobs fanning out tasks.
- The backlog is what matters, meaning your real objective is "messages processed within N minutes", not request latency.
- Consumers mostly wait on I/O, so resource metrics stay flat regardless of how much work is pending.
Who should use KEDA queue depth autoscaling?
KEDA queue depth autoscaling is for teams that already run their RabbitMQ consumers on Kubernetes and want replica counts to follow the backlog without manual intervention. In practice that usually means:
- Platform and SRE teams who run many worker Deployments and want one consistent scaling pattern instead of per-service scripts or fixed replica counts.
- Application teams that own consumer services and can describe their throughput as "a single replica handles roughly X messages per second". That number is what you need to set a sensible per-replica target.
- Teams with idle periods worth reclaiming, such as batch pipelines, internal tooling queues, or staging and preview environments where workers can sit at zero most of the day.
The consumers themselves must be suited to horizontal scaling:
- They follow the competing consumers pattern: several replicas read from the same queue, and any replica can process any message.
- They are stateless between messages, or keep their state in an external store, so a pod can be added or removed at any time.
- They can shut down gracefully and process messages idempotently, because scale-down and redelivery are routine events, not exceptions.
It is a poor fit when only one consumer may process the queue at a time, for example when strict ordering is required or the queue uses RabbitMQ's single active consumer feature. In that case extra replicas sit idle and add no throughput. It also does little for a consumer whose throughput is capped by a downstream dependency rather than by the number of replicas. The last section covers alternatives for those cases.
How does queue depth autoscaling work, end to end?
The full loop has a small number of moving parts:
- Producers publish messages to a RabbitMQ queue. Consumer pods read from that queue and ack each message after processing.
- KEDA reads the queue on a fixed interval. The RabbitMQ scaler connects either over AMQP or through the management HTTP API and reads the queue's message count (or message rate, depending on
mode). - KEDA handles the 0 ↔ 1 step itself. If the workload is at zero and the queue length exceeds the activation threshold, KEDA scales it to one replica. If the trigger stays inactive for the cooldown period, KEDA scales it back to zero.
- The HPA handles 1 ↔ N. KEDA exposes the queue metric through the Kubernetes external metrics API. The HPA that KEDA generates divides the metric by the per-replica target and sets the Deployment's replica count, within the configured minimum and maximum.
- New pods start consuming, the backlog drains, and the metric falls.
- The HPA scales down as the metric drops, subject to its stabilization window and scale-down policies. During scale-down, terminating pods stop consuming and finish or requeue their in-flight messages.
The rest of this article covers each part in detail: how KEDA and the HPA divide the work, how to write the ScaledObject, how the target becomes a replica count, how scale-to-zero and flapping behave, and what consumers need to do to make scale-down safe.
How does KEDA work with the Kubernetes HPA?
KEDA doesn't replace the Horizontal Pod Autoscaler. It configures and feeds one. When you apply a ScaledObject, the KEDA operator reads it and creates an ordinary HPA for the target Deployment or StatefulSet, then keeps that HPA in sync with the ScaledObject spec. You can inspect it with kubectl get hpa. Its name is derived from the ScaledObject, and you shouldn't edit it by hand, because the operator will overwrite your changes.
The HPA can't query RabbitMQ directly. KEDA's metrics adapter registers itself as the provider of the Kubernetes external metrics API. On each HPA sync, the adapter asks the relevant scaler for its current value (queue length or message rate) and returns that value as an external metric. The HPA then applies its usual replica calculation.
The HPA doesn't scale a workload to zero by default. KEDA covers that range itself. The operator polls each trigger on the ScaledObject's polling interval. When the queue is empty, it scales the workload to zero. When work appears, it activates the workload back to one replica. Above one replica, the HPA takes over. This creates a delay: a message arriving at an idle queue can wait up to one polling interval, plus pod startup time, before anything consumes it.
This design makes KEDA part of your control plane. If the operator or the metrics adapter goes down, the HPA can no longer fetch metrics and stops scaling, and scaled-to-zero workloads stay at zero. Alert on the health of both components and on HPA conditions that report metric fetch failures.
Don't add a second HPA to the same workload. Two HPAs targeting the same workload act as competing control loops and overwrite each other's replica counts. If you also want to scale on CPU, add a cpu trigger to the ScaledObject. The HPA that KEDA generates then evaluates all triggers and uses the highest resulting replica count.
+----------------+ +-----------------------+
| ScaledObject |--watched-by->| KEDA operator |
| triggers: | | creates + syncs HPA |
| rabbitmq, cpu | | polls each trigger |
+----------------+ | scales 0 <-> 1 |
+-----------------------+
| |
creates/syncs | | activate / idle
v | (0 <-> 1)
+-----------------------+ +--------------+ |
| KEDA metrics adapter |<---| HPA (by KEDA)| |
| external metrics API | | fetch metric | |
+-----------------------+ +--------------+ |
| | |
scaler | scale 1..N | |
query v v v
+-----------------------+ +---------------------------+
| RabbitMQ | | Deployment / StatefulSet |
| queue length or rate | | RabbitMQ consumer pods |
+-----------------------+ +---------------------------+
! No second HPA on this workload: loops conflict. Use a cpu trigger.
! Operator/adapter down => HPA stops scaling; zero stays at zero.
! Idle queue: first message waits <= pollingInterval + pod startup.
The KEDA operator turns a ScaledObject into a managed HPA and handles 0-to-1 activation itself, while the HPA scales from 1 to N using queue metrics that KEDA's metrics adapter serves through the external metrics API.
How do you configure a ScaledObject for RabbitMQ scaling?
A ScaledObject names the workload to scale and the trigger that drives it. Work through it from the top.
scaleTargetRef points at the consumer Deployment by name. KEDA creates and owns the HPA for that target, so delete any HPA you wrote by hand for it.
minReplicaCount and maxReplicaCount set the replica floor and ceiling. Base the ceiling on what downstream systems can absorb, such as database connections and third-party rate limits, rather than on cluster capacity. A broker backlog can easily ask for more consumers than your database can serve.
pollingInterval sets how often, in seconds, KEDA queries the queue. This is a trade-off. If the trigger reads queue stats over HTTP, every poll is a request to RabbitMQ's management API. Short intervals across many ScaledObjects add steady load to a component that is not built for high request rates. Long intervals let a burst sit unprocessed until the next check.
cooldownPeriod applies only to the final step from one replica to zero. Section 5 covers this in detail.
The rabbitmq trigger's metadata has three fields:
queueName: the queue to measure.mode: set it toQueueLength, which scales on queued messages. The alternative,MessageRate, scales on publish rate.value: the target number of messages per replica.
Do not put the broker URL, which includes credentials, in the trigger metadata. Store it in a Kubernetes Secret. Then create a TriggerAuthentication whose secretTargetRef maps the trigger's host parameter to a key in that Secret. The trigger references this through authenticationRef. With this setup, you can rotate credentials without editing the ScaledObject, and the URL stays out of manifests in version control.
apiVersion: v1
kind: Secret
metadata:
name: rabbitmq-conn
type: Opaque
stringData:
# Management API URL with credentials and vhost path. Placeholder values only.
host: ""
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: rabbitmq-auth
spec:
secretTargetRef:
- parameter: host # fills the trigger's host parameter from the Secret
name: rabbitmq-conn
key: host
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-consumer
spec:
scaleTargetRef:
name: order-consumer # the Deployment; KEDA creates and owns its HPA
minReplicaCount: 0
maxReplicaCount: 20 # cap at what the DB pool can serve, not cluster size
pollingInterval: 30 # each poll is one request to the management API
cooldownPeriod: 300 # applies only to the final 1 -> 0 step
triggers:
- type: rabbitmq
metadata:
protocol: http # read queue stats via the management API
queueName: orders
mode: QueueLength # scale on messages waiting in the queue
value: "50" # target messages per replica
authenticationRef:
name: rabbitmq-auth # no credentials in this manifest
A Secret, a TriggerAuthentication and a ScaledObject that scale a RabbitMQ consumer Deployment on queue length, while keeping the broker URL and credentials out of the ScaledObject.
How does the target value become a replica count?
The HPA does not track individual messages. KEDA reports the queue length as an external metric with an average-value target, and the HPA computes roughly ceil(queue length / target per replica). It then clamps the result between minReplicaCount and maxReplicaCount. The HPA also ignores small deviations within its tolerance band, which stops it from adding or removing a pod for every few messages.
Illustrative estimate, using assumed inputs:
| Input | Value | Result |
|---|---|---|
| Queue length | 1,200 messages | — |
| Target per replica | 100 | ceil(1,200 / 100) = 12 replicas |
maxReplicaCount = 20 |
— | 12 replicas |
maxReplicaCount = 8 |
— | capped at 8 replicas |
Set the target from your own measurements. Use target ≈ per-replica processing rate × acceptable drain time. If a single consumer handles 5 messages per second and you can tolerate a 20-second backlog per replica, the target is 100. If you lower the target, the deployment scales more aggressively. If you raise it, you get fewer pods and more latency.
Keep the target consistent with the consumer prefetch setting. Depending on how the scaler reads the queue, messages that have been delivered but not yet acknowledged may not count as "ready." Suppose each of 8 consumers prefetches 200 messages. Up to 1,600 messages can then sit in their buffers while the queue looks almost empty, and the HPA scales down while real work is still waiting. To avoid this, keep the prefetch small compared with the target.
Choose maxReplicaCount based on what your dependencies can absorb, not only on cost. Every replica opens database connections and calls downstream APIs. If you allow unlimited fan-out against a backlog, the queue problem can turn into an outage in a downstream system.
Should you scale to zero, and how do you prevent flapping?
Setting minReplicaCount: 0 lets the deployment drop to zero pods when the queue is empty. The HPA does not handle that range. The KEDA operator polls the trigger and scales the workload from zero to one replica itself when the metric crosses the activation threshold (for RabbitMQ, activationValue in the trigger metadata). After that, the HPA manages scaling from one replica upward. The reverse step to zero waits for cooldownPeriod, measured from the last time the trigger was active.
Scaling from zero is slow for the first message. Before it can be acked, the cluster has to:
- schedule a pod, possibly on a node the cluster autoscaler has not provisioned yet,
- pull the image if the node does not have it cached,
- start the runtime,
- open an AMQP connection and channel.
The polling interval adds to that. This delay is fine for nightly batch work. It is not fine for sparse traffic that someone is waiting on, such as a password-reset email or a webhook relay. For those queues, set minReplicaCount: 1. A single idle consumer usually costs less than a stalled user flow.
Flapping comes from bursty queues: a backlog spikes, replicas scale up, they drain it in seconds, and the HPA scales down just before the next burst arrives. You can tune this under advanced.horizontalPodAutoscalerConfig.behavior:
- Set
scaleDown.stabilizationWindowSecondsto cover the gap between bursts. The HPA then uses the highest recommendation within that window, so it holds capacity through short lulls. - Add a
scaleDownpolicy that limits removal per period, for example a fixed number of pods or a percentage. Replicas then shrink gradually instead of all at once. - Keep
scaleUpresponsive, with a short or zero window, so backlog growth still triggers scale-up quickly.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: orders-consumer
spec:
scaleTargetRef:
name: orders-consumer # Deployment running the RabbitMQ consumer
pollingInterval: 15 # how often KEDA checks the queue (adds to cold-start delay)
cooldownPeriod: 300 # wait 5 min after the trigger was last active before going 1 -> 0
minReplicaCount: 0 # batch-style queue; use 1 for sparse, latency-sensitive traffic
maxReplicaCount: 20
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # hold capacity through lulls between bursts
policies:
- type: Pods
value: 2 # remove at most 2 pods...
periodSeconds: 60 # ...per minute
scaleUp:
stabilizationWindowSeconds: 0 # react to backlog growth immediately
policies:
- type: Percent
value: 100 # allow doubling replicas...
periodSeconds: 15 # ...every 15 seconds
triggers:
- type: rabbitmq
metadata:
protocol: amqp
queueName: orders
mode: QueueLength
value: "50" # target messages per replica (HPA, 1..N)
activationValue: "0" # KEDA activates 0 -> 1 when queue length exceeds this
authenticationRef:
name: rabbitmq-auth # TriggerAuthentication holding the AMQP host URI
A KEDA ScaledObject for a RabbitMQ consumer that scales to zero with an activation threshold and cooldown, and uses HPA behavior settings to scale up quickly while scaling down slowly to avoid flapping on bursty queues.
Graceful shutdown, idempotency and debugging
Scale-down only reduces capacity safely if the pods being removed don't lose or corrupt work. When the HPA lowers the replica count, Kubernetes sends SIGTERM to the chosen pods, waits terminationGracePeriodSeconds (30 seconds by default), then sends SIGKILL. The consumer should use that window as follows:
- Stop consuming first. Cancel the consumer so RabbitMQ stops delivering new messages to this pod.
- Drain in-flight work. Finish and ack the messages already received. If a message can't finish in time, nack it with requeue so another replica picks it up.
- Size the grace period to the work. Base
terminationGracePeriodSecondson your prefetch count multiplied by your slowest realistic handler time, plus some margin. A low prefetch keeps that upper bound small.
If the process dies with unacked messages, RabbitMQ requeues them when the channel closes. Nothing is lost, but some messages will be delivered twice. This is why handlers must be idempotent. Key each side effect on a stable message ID, such as a unique constraint, a processed-IDs table, or an upsert. Treat the redelivered flag as a hint only, not a guarantee.
Verifying the scaling path
kubectl get hpalists the HPA that KEDA created for the ScaledObject. Check that current and target metric values are populated rather than<unknown>.kubectl describe hpashows theAbleToScale,ScalingActiveandScalingLimitedconditions, plus recent events. "Failed to get external metric" means the metric pipeline is broken.kubectl get scaledobjectshowsREADYandACTIVE. Usedescribeto read the condition messages.- If the metric is missing or stale, read the KEDA operator and metrics API server logs in KEDA's namespace. Look for authentication failures, unreachable RabbitMQ endpoints, or a wrong queue name.
#!/usr/bin/env bash
NS=orders
SO=orders-consumer
kubectl get hpa "keda-hpa-${SO}" -n "$NS"
kubectl describe hpa "keda-hpa-${SO}" -n "$NS"
kubectl get scaledobject "$SO" -n "$NS"
kubectl describe scaledobject "$SO" -n "$NS" # read the condition messages
kubectl get apiservice v1beta1.external.metrics.k8s.io
kubectl logs deploy/keda-operator -n keda --since=15m | grep -i -E "error|${SO}"
kubectl logs deploy/keda-operator-metrics-apiserver -n keda --since=15m | grep -i error
kubectl get pods -n "$NS" -w
A shell checklist that traces a stuck or missing scaling decision from the KEDA-created HPA through ScaledObject conditions, the external metrics APIService and the KEDA operator and metrics API server logs, then watches pod termination during scale-down.
When should you avoid KEDA queue depth autoscaling, and what are the alternatives?
Queue length measures backlog. It does not tell you how long a message waits or what processing it costs. Suppose 500 messages each take 10 ms and another 500 each take 30 seconds. Both queues have the same length, but the second needs far more consumers. If processing times vary widely, a fixed per-replica target either over-provisions on cheap messages or falls behind on expensive ones. Queue depth is also a poor fit when your SLO is end-to-end latency, or when the real bottleneck is downstream, such as a database connection pool. In that case more consumers only add contention.
| Approach | Best fit | Main drawback |
|---|---|---|
| CPU/memory HPA | CPU-bound consumers with uniform work | Lags behind backlog; I/O-bound consumers stay idle while the queue grows |
| HPA on custom metrics via a metrics adapter | Teams already exporting metrics such as message age | You build and run the adapter and its query configuration |
| Rate-based or custom signals (e.g. publish rate minus ack rate, oldest message age) | Variable processing times, latency SLOs | Requires instrumentation and more careful tuning |
| Fixed replica count | Steady, predictable load | Pays for peak capacity at all times |
On engineering cost, a single declarative ScaledObject replaces hand-written scaling scripts and most manual capacity tuning, and it is reviewed like any other manifest. In exchange, you run, upgrade and monitor the KEDA operator as part of your control path. Your consumers must also tolerate frequent scale-down, which means engineering time spent on graceful shutdown and idempotent handlers. Without that work, autoscaling produces duplicated or lost messages.
How can KEDA queue depth autoscaling save engineering time?
Most of the time savings come from removing manual and bespoke work around capacity:
- No manual scaling during backlogs. Without autoscaling on the queue, someone has to notice a growing backlog, decide how many replicas to add, run
kubectl scaleor edit a manifest, and remember to scale back down afterwards. With a ScaledObject, the HPA does both steps, and on-call engineers only step in when the system hitsmaxReplicaCountor a downstream limit. - No custom scaling controllers or cron jobs. Teams often write small scripts that poll the RabbitMQ management API and patch replica counts. Those scripts need credentials, error handling, retries, and someone to maintain them. KEDA's RabbitMQ scaler and the standard HPA replace that code with configuration.
- One pattern reused across queues. Once the Secret, TriggerAuthentication and a reviewed ScaledObject exist, adding autoscaling to another consumer is mostly a matter of copying the manifest and changing
scaleTargetRef,queueName,valueand the replica bounds. The same debugging checklist applies to every queue. - Less up-front capacity planning. Instead of sizing each worker Deployment for its worst burst, you measure per-replica throughput once, set a target and a ceiling, and let replicas follow demand. Changes to traffic patterns usually mean adjusting a number in a manifest rather than re-planning capacity.
- Changes go through normal review. Scaling behavior lives in version control next to the Deployment, so it is reviewed, diffed and rolled back like any other change, rather than living in someone's shell history.
Illustrative estimate, using assumed inputs: suppose a team runs 6 consumer Deployments, and during bursty periods on-call engineers manually scale one of them about twice a week, spending an assumed 20 minutes each time to notice the backlog, scale up, watch it drain and scale back down. That is roughly 40 minutes a week of reactive work that the ScaledObject takes over. Set this against the one-time cost of adding graceful shutdown and idempotent handlers to each consumer, and the ongoing cost of upgrading and monitoring KEDA. Substitute your own incident history to see whether the trade-off holds for your team.
The savings depend on the prerequisites in the previous sections. If consumers cannot shut down cleanly or process duplicates safely, the time saved on manual scaling is spent investigating lost or repeated work instead.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to keda queue depth autoscaling, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.