Why do CPU and memory fail as scaling signals for queue workers?
Queue-depth autoscaling sets the number of worker replicas from the amount of work waiting in a broker or job queue, not from how hard existing pods are working. This guide is for backend and platform engineers who run asynchronous consumers on Kubernetes, such as workers that read from a message broker, a task queue, or a database-backed job table.
CPU and memory are poor signals for these workloads because they measure effort, not demand. A typical worker spends most of its time waiting on network calls to a database, an external API, or object storage. It can sit at 15% CPU while the queue grows by thousands of messages a minute, so a CPU-based HPA sees no reason to act. When CPU does rise, it is a lagging indicator. The backlog has already built up, new pods still need to be scheduled, pull images, and warm connections, and latency for queued jobs keeps climbing in the meantime. Memory is worse, since many runtimes hold onto allocated heap whether or not there is work to do.
Queue depth measures demand directly, but raw queue length is not the right target on its own. Ten thousand messages is a crisis for a worker that handles five per second and routine for one that handles five hundred. What matters is backlog per replica, or the estimated time the current fleet needs to drain the queue. The rest of this guide builds on that idea.
Who should use queue-depth autoscaling?
Queue-depth autoscaling fits teams whose workers meet most of these conditions:
- Consumers pull work from a queue. The worker reads from a broker, a task queue, or a job table, and the backlog can be measured from outside the workers.
- Load varies. Arrivals come in bursts, follow daily cycles, or depend on upstream batch jobs, so a fixed replica count is either too small at peak or wasteful at idle.
- Queued work has a latency expectation. You care how long a message waits before it is processed, and you can state an acceptable drain time.
- Per-message cost is roughly uniform. Most messages take a similar amount of time, so message count is a reasonable proxy for total work.
- Workers are stateless and safe to stop. Replicas can be added and removed without manual coordination, and consumers tolerate redelivery.
It is a poor fit for synchronous request-serving services, where request rate or latency is the better signal, and for workloads with steady, predictable load, where fixed replicas are simpler. The last major section covers these cases in more detail.
On the platform side, you need either a Prometheus stack with an exporter for your broker (for the Prometheus Adapter route) or the ability to install and operate the KEDA operator in the cluster.
How does queue-depth autoscaling work?
The control loop has five steps:
- Measure. A component reads the number of waiting messages from the broker. With the Prometheus Adapter, an exporter publishes it as a Prometheus gauge. With KEDA, a built-in scaler queries the broker directly.
- Expose. That component serves the value through the
external.metrics.k8s.ioAPI, which the Kubernetes API server aggregates. - Compute. The HPA controller reads the metric on each sync and divides the backlog by a per-replica target to get a desired replica count, bounded by
minReplicasandmaxReplicas. - Scale. The HPA updates the Deployment's scale subresource. New pods start consuming, or surplus pods are terminated.
- Feed back. As workers drain the queue, depth falls, the next sync computes fewer replicas, and the HPA scales down after its stabilization window.
KEDA adds one step outside the HPA: it activates the Deployment from zero to one replica when messages appear, and returns it to zero after the queue has been empty for a cooldown period. The HPA cannot do this on its own. The next section explains the HPA side of the loop in detail.
How does the HPA scale on custom and external metrics?
The HPA controller does not talk to your broker. It queries metrics APIs that the API server aggregates, and some other component has to serve them. Two of these APIs matter here:
custom.metrics.k8s.ioserves metrics attached to Kubernetes objects, such as a per-pod requests-per-second value or a value on an Ingress.external.metrics.k8s.ioserves metrics with no Kubernetes object behind them, such as queue length or consumer lag.
Queue depth belongs to the broker, not to any pod. If every worker reported the same backlog as a pod metric, the HPA would average identical numbers and gain nothing. An External metric states the real situation: one number per queue, read from outside the cluster.
broker ──> exporter / scaler ──> metrics API ──> HPA controller ──> Deployment
(queue) (reads depth) (external. (polls, computes (scale
metrics.k8s.io) desiredReplicas) subresource)
On each sync, the controller computes:
desiredReplicas = ceil(currentReplicas × currentValue / targetValue)
With an AverageValue target on an External metric, the controller first divides the metric by the current replica count. The target therefore means "messages per replica." In this illustrative example, the queue holds 1,200 messages, 4 replicas are running, and the target is 100:
- The per-replica value is 1,200 / 4 = 300.
- desiredReplicas = ceil(4 × 300 / 100) = 12.
The replica counts cancel out, so the result is ceil(total / target). The HPA sizes the fleet from the backlog directly. A tolerance band around the target, 10% by default, prevents scaling on small changes.
+------------+ +--------------+ +-------------------------+
| Broker |---->| Exporter / |---->| kube-apiserver |
| |depth| scaler | | aggregation layer |
| queue=1200 | | (metrics | | external.metrics.k8s.io |
+------------+ | adapter) | +------------+------------+
+--------------+ |
| HPA reads value = 1200
v
+--------------+ +---------------+----------------+
| Deployment |<--------------------| HPA controller (each sync) |
| workers | replicas: 12 | AverageValue target = 100 |
| 4 -> 12 | | per-replica = 1200 / 4 = 300 |
+--------------+ | ceil(4 x 300 / 100) = 12 |
| ratio 3.0: outside 10% band |
+--------------------------------+
Queue depth flows from the broker through a scaler that serves external.metrics.k8s.io via the API server's aggregation layer, and the HPA controller turns an AverageValue target of 100 into ceil(1200 / 100) = 12 replicas on the Deployment's scale subresource (illustrative numbers).
How do you choose a per-replica queue depth target?
The target is the backlog one replica should own: target = per-worker throughput × acceptable drain time. If each worker clears its share within the drain time, the whole fleet does too, because the HPA sizes replicas to total backlog divided by target. Measure throughput under production-like conditions, with real payloads and real downstream latency. A synthetic benchmark against an idle database will overstate it, and the autoscaler will under-provision.
Worked example (illustrative estimate)
Assumed inputs:
- Each worker sustains 5 messages/second.
- The SLO allows a backlog to drain in 60 seconds.
- Peak backlog is 9,000 messages.
The target is 5 × 60 = 300 messages per replica. At peak, the HPA asks for 9,000 / 300 = 30 replicas.
Cap maxReplicas by what's downstream
Queue size tells you how many workers you want. It does not tell you how many your dependencies can serve. Suppose each worker holds 4 database connections and the pool budget for this service is 100. That limits you to 25 replicas. Suppose a third-party API allows 100 requests/second and each message makes one call. That limits you to 100 / 5 = 20 replicas.
Set maxReplicas to the lowest of these limits, which is 20. At 20 replicas, the fleet drains 9,000 messages in 90 seconds instead of 60. That is a slower drain. Without the cap, the fleet would exhaust the connection pool or trip the rate limiter, and throughput would collapse.
Leave headroom for slow capacity
New replicas don't appear instantly. A node may need to be provisioned and an image pulled before a pod processes anything. To absorb that lag, keep minReplicas above zero, or set the target somewhat below the computed value so scale-out starts earlier.
Implementing queue depth scaling with the Prometheus Adapter
Start by getting queue depth into Prometheus as a gauge. Use an existing exporter for your broker, or have a small sidecar poll the queue and expose something like queue_messages_ready{queue="orders"}. Scrape the broker's view of the queue, not counts reported by workers. Worker-side metrics disappear when workers scale down, and that is exactly when you need the signal.
Expose it through externalRules
In the adapter config, an externalRules entry needs three things: a seriesQuery that finds the series, a name that sets the metric name the HPA requests, and a metricsQuery that computes the value. Wrap the query in sum(...) by (queue). If several exporter replicas or partitions report the same queue, an unaggregated query returns multiple series, and the HPA gets a value you did not intend.
Three rule mistakes account for most failures:
- Label mismatch. The HPA's metric selector must match labels that actually exist on the series.
- Wrong aggregation. Using
avgormaxinstead ofsummisreports the total backlog. - Namespace scoping. The adapter adds the HPA's namespace as a label matcher by default. If the exporter runs in another namespace, the query silently returns nothing.
Verify before trusting the HPA
Query /apis/external.metrics.k8s.io/v1beta1/namespaces/<ns>/<metric> with kubectl get --raw and confirm the response contains a single item with a plausible value. An empty list here explains a <unknown> target in kubectl describe hpa.
Write the HPA
Then write an autoscaling/v2 HPA with an External metric and an AverageValue target. The HPA divides the total backlog by the current replica count and compares the result to your per-replica target.
Limitations
Every reading has passed through a scrape interval, the adapter's query, and the HPA sync period, so the autoscaler reacts later than a direct broker query would. The adapter also cannot scale a deployment to zero by itself. The HPA's minReplicas floor still applies, so idle queues keep at least one worker running.
externalRules:
- seriesQuery: 'queue_messages_ready{queue!=""}'
resources:
# Maps the series' namespace label to the HPA's namespace. The exporter's
# series must carry the HPA's namespace, or the query returns nothing.
overrides:
namespace: {resource: "namespace"}
name:
matches: "^queue_messages_ready$"
as: "orders_queue_depth" # name the HPA requests
# sum, not avg/max: total backlog across exporter replicas or partitions
metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (queue)'
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: orders-worker
namespace: workers
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: orders-worker
minReplicas: 1 # the adapter alone cannot scale to zero
maxReplicas: 20
metrics:
- type: External
external:
metric:
name: orders_queue_depth
selector:
matchLabels:
queue: orders # must match a label that exists on the series
target:
type: AverageValue
averageValue: "50" # target messages per replica (total / replicas)
A Prometheus Adapter externalRules entry that exposes summed queue depth as an external metric, the kubectl get --raw check to run first, and an autoscaling/v2 HPA that targets an AverageValue of 50 messages per replica.
Implementing queue depth scaling with KEDA
KEDA removes that floor problem by querying the broker directly and managing the HPA itself. You define a ScaledObject that points at your worker Deployment through scaleTargetRef and lists one or more triggers. For RabbitMQ, the trigger names the queue, uses queue-length mode, and sets a target value per replica, such as 20 messages. Broker credentials do not belong in the ScaledObject. Put them in a TriggerAuthentication that maps a Kubernetes Secret key, such as the connection string, to the scaler's host parameter, and reference it from the trigger with authenticationRef. This lets several ScaledObjects share one credential definition, and it keeps secrets out of manifests that application teams edit.
KEDA's operator then creates an HPA for the Deployment. Do not create your own HPA alongside it, because the two would fight over the replica count. Scaling happens in two separate phases:
- 0 to 1 (activation): KEDA handles this itself. When the queue depth exceeds
activationValue, it activates the Deployment. When the depth stays below that value throughcooldownPeriod, it scales the Deployment back to zero. - 1 to N (scaling): The HPA handles this, using the target
value.
You can set the activation threshold low, so any message wakes a worker, while the per-replica target stays high.
KEDA has operational costs. It is another operator with its own metrics API server and CRDs. You have to upgrade it, monitor it, and grant it RBAC permissions. It also needs network access to your brokers. A cluster can register only one provider for the external metrics API, so KEDA and an external-metrics Prometheus Adapter cannot both serve it.
Scale-to-zero also adds latency on the first message. That delay is the polling interval plus pod scheduling, image pull, and application startup. For sporadic work with a latency deadline, set minReplicaCount to 1.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: rabbitmq-auth
namespace: workers
spec:
secretTargetRef:
- parameter: host # maps to the rabbitmq scaler's 'host' param
name: rabbitmq-conn # Kubernetes Secret name
key: host # key inside the Secret
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-worker
namespace: workers
spec:
scaleTargetRef:
name: order-worker # Deployment; KEDA creates its HPA, so do not add your own
pollingInterval: 15 # seconds between broker checks (part of cold-start delay)
cooldownPeriod: 300 # seconds below activation before scaling back to 0
minReplicaCount: 0 # allow scale-to-zero
maxReplicaCount: 30
triggers:
- type: rabbitmq
metadata:
protocol: amqp
queueName: orders
mode: QueueLength # scale on messages in the queue
value: "20" # 1-to-N: HPA target of ~20 messages per replica
activationValue: "0" # 0-to-1: any message (>0) wakes a worker
authenticationRef:
name: rabbitmq-auth # shared credential definition, no secrets here
A TriggerAuthentication that pulls the RabbitMQ connection string from a Secret, and a ScaledObject that uses it to activate the worker Deployment from zero on any message and lets the KEDA-managed HPA scale at about 20 messages per replica.
What are the alternatives to queue-depth autoscaling?
Scaling a long-running Deployment on queue depth is one option among several. The main alternatives are:
- HPA on CPU or memory. This needs no extra components and works when workers are CPU-bound and CPU tracks backlog closely. For I/O-bound consumers it reacts late or not at all, for the reasons in the first section.
- Fixed replica count. Size the Deployment for peak load and leave it. This is the simplest option to operate and the right choice for steady load, at the cost of idle capacity off-peak.
- Scheduled scaling. If load follows a known calendar, such as business hours or a nightly batch, change replica counts on a schedule. You can do this with a CronJob that runs
kubectl scale, or with KEDA's cron scaler. It handles predictable peaks but not unexpected bursts. - One Job per unit of work. Instead of scaling long-running consumers, start a Kubernetes Job for each message or batch. KEDA's
ScaledJobdoes this from queue depth. It suits long-running, variable-duration tasks, because a Job is not killed mid-task by a scale-down, but it pays pod startup cost on every unit of work. - Tuning concurrency inside the worker. Raising prefetch counts, thread pools, or async concurrency in each consumer can increase throughput without adding pods, as long as the bottleneck is waiting on I/O rather than CPU or a downstream limit.
- Other custom metrics. Message age, consumer lag, or processing latency can replace or supplement depth. The next section describes using message age as a second trigger.
These are not mutually exclusive. A common arrangement combines a scheduled floor for known peaks with queue-depth scaling for bursts above it.
Should you use KEDA or the Prometheus Adapter, and when should you avoid queue-based autoscaling?
| Option | Scale-to-zero | Broker integration | Latency | Operational overhead | Credentials required | Fit with existing Prometheus |
|---|---|---|---|---|---|---|
| Native HPA on CPU | No | None | Lags the backlog | Lowest | None | Not needed |
| Prometheus Adapter | No | Via exporters | Scrape + query + HPA sync | Adapter, rules, exporters | Exporter only | Strong |
| KEDA | Yes | Built-in scalers | Polling interval | KEDA operator, CRDs | KEDA reads the broker | Optional |
| Fixed or scheduled replicas | Scheduled only | None | None, if sized well | Low | None | Not needed |
The cost trade-offs differ by option:
- Prometheus Adapter. If you already run Prometheus and exporters, this adds little to operate. Development cost goes into writing and testing the metric rules.
- KEDA. Each scaler is a small manifest, so development is cheap. The cost moves to operations: you run another controller, and broker credentials have to be stored where KEDA can read them.
- Fixed replicas. These cost nothing to build. You pay instead for idle capacity.
When depth-only scaling misleads
Avoid scaling on depth alone in these cases:
- Mixed workloads. If one message takes 10 ms and another takes 10 minutes, a single depth target cannot express the actual amount of work in the queue.
- Fragile downstream systems. Adding consumers can exhaust a database connection pool or trip rate limits. Cap
maxReplicasat what the dependency can tolerate, or do not autoscale. - Steady, predictable load. Fixed or scheduled replicas avoid scaling churn and are easier to reason about.
- Sporadic, latency-critical work. If a cold start is unacceptable, keep at least one warm replica.
Add message age or consumer lag as a second trigger. When several metrics are configured, the HPA scales to the highest replica count any of them proposes. A slow trickle of expensive messages, or a stuck consumer with a short queue, will still push oldest-message age up and cause a scale-up.
How do you tune scaling behavior, shut down workers safely, and monitor the autoscaler?
That signal is only useful if the autoscaler reacts to it predictably and the workers it removes leave cleanly.
Tune behavior for fast ramp-up and slow ramp-down
Set scaleUp.stabilizationWindowSeconds to zero or a few seconds so a backlog triggers new replicas immediately. On scale-down, use a longer stabilization window and a conservative policy, such as removing a fixed number of pods or a small percentage per period. Queue depth is spiky: a batch arrives, workers drain it, and depth drops to zero. Without a scale-down window, the HPA cuts replicas just before the next batch and then has to scale back up, which wastes pod startup time.
Shut down workers without losing or duplicating work
When a pod receives SIGTERM, it should stop pulling new messages and either finish its in-flight jobs or nack them so another worker picks them up. Set terminationGracePeriodSeconds longer than your longest job, because the kubelet sends SIGKILL when that period ends. Acknowledge a message only after its work completes. Delivery is still at-least-once, so make consumers idempotent.
Watch the autoscaler itself
kubectl describe hpa shows the status conditions and recent events. AbleToScale reports whether the HPA can change replicas. ScalingActive turns false when the HPA cannot fetch the metric. ScalingLimited means it has hit the min or max replica bound. Alert when the metric is missing or stale, because the HPA stops acting on a metric it cannot fetch. That failure is silent: the deployment stays at its current size while the backlog grows.
These practices reduce manual replica changes, scaling-related pages, and cleanup after messages are processed twice.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: order-worker
spec:
scaleTargetRef:
name: order-worker # Deployment to scale
minReplicaCount: 1
maxReplicaCount: 30
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # react to a backlog immediately
policies:
- type: Percent
value: 100 # at most double replicas...
periodSeconds: 15 # ...every 15s
scaleDown:
stabilizationWindowSeconds: 300 # use the highest recommendation of the last 5 min
policies:
- type: Pods
value: 2 # remove at most 2 pods...
periodSeconds: 60 # ...per minute
triggers:
- type: rabbitmq
metadata:
queueName: orders
mode: QueueLength
value: "20" # target messages per replica
hostFromEnv: RABBITMQ_HOST # connection string from the worker's env
---
A KEDA ScaledObject that scales up immediately and scales down slowly on RabbitMQ queue depth, with a grace period sized to the longest job and the command to inspect the HPA KEDA generates.
How can queue-depth autoscaling save engineering time?
Without demand-based scaling, someone has to watch backlogs and change replica counts by hand, or the team over-provisions and revisits sizing whenever load patterns change. Queue-depth autoscaling moves that work into a control loop. The time savings come from a few places:
- Fewer manual scaling interventions. Engineers no longer run
kubectl scaleor edit replica counts during bursts, and nobody needs to remember to scale back down afterward. - Fewer backlog pages. Alerts for a growing queue fire less often when capacity follows demand, and those that do fire are more likely to point to a real fault, such as a stuck consumer or a failing dependency, rather than ordinary load.
- Less capacity-planning rework. The per-replica target is derived from per-worker throughput and drain time. When load grows, the replica count follows without a new sizing exercise. You revisit the target only when per-message cost or the SLO changes.
- Faster diagnosis. With
maxReplicasset from downstream limits and alerts on missing metrics, an incident shows up as a specific condition (ScalingLimited,ScalingActive=False) instead of a vague "workers are slow."
Worked example (illustrative estimate)
Assumed inputs:
- Before autoscaling, on-call engineers adjust worker replicas by hand about 3 times per week.
- Each adjustment, including noticing the backlog, scaling, and later scaling back down, takes about 20 minutes.
- Initial setup of the exporter or KEDA scaler, the target calculation,
behaviortuning, and alerts takes about 2 engineer-days (16 hours).
Manual work is 3 × 20 = 60 minutes, or about 1 hour per week. Under these assumptions, the setup cost is recovered in roughly 16 weeks, before counting reduced over-provisioning or interruptions outside working hours. With different inputs the result changes: a team that scales by hand once a month may never recover the setup cost, while a team running many worker Deployments recovers it faster, because the same pattern and manifests are reused across services.
Autoscaling also adds work of its own: maintaining the adapter or KEDA, keeping exporters healthy, and reviewing targets when workloads change. Count that ongoing cost against the savings.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to hpa custom metrics queue depth, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.