HPA Custom Metrics with KEDA: Scale Kubernetes on Real Work

What HPA custom-metric scaling with KEDA is

The Horizontal Pod Autoscaler (HPA) is the built-in Kubernetes controller that changes the replica count of a Deployment, StatefulSet or other resource with a scale subresource. Out of the box it scales on CPU and memory. Custom-metric scaling points the same controller at a different signal, one that measures the work waiting to be done. Examples include requests per second per pod, in-flight requests, queue length or consumer-group lag.

Two components supply that signal:

• The Prometheus Adapter exposes metrics that already exist in Prometheus through the Kubernetes custom and external metrics APIs, so a hand-written HPA can consume them.
• KEDA (Kubernetes Event-driven Autoscaling) is an operator plus a metrics server. You declare a ScaledObject or ScaledJob that names a workload and one or more triggers, such as a RabbitMQ queue, a Kafka consumer group or a Prometheus query. KEDA then generates and manages the HPA for you. It also adds scale-to-zero, which the HPA cannot do on its own.

KEDA does not replace the HPA. It configures the HPA and feeds it metrics. Scaling from one replica to many is still done by the standard HPA controller.

Why CPU is the wrong scaling signal and how the HPA computes replicas

CPU utilization only tells you how busy a pod's processors are, and that is often not how much work is waiting. A queue consumer that spends most of its time waiting on a database or downstream API can sit at 20% CPU while its backlog grows by thousands of messages. A request-bound API can breach its latency target because requests are queueing on connection pools or threads well before CPU saturates. In both cases a CPU-driven autoscaler reacts late, or not at all. Custom-metric autoscaling replaces CPU with a signal that tracks the pending work directly, such as queue depth, in-flight requests or requests per second. This matters most to engineers running consumers, background workers and latency-sensitive APIs.

The HPA control loop applies one formula to whatever metric you choose:

desired = ceil(currentReplicas × currentMetric / targetMetric)

Illustrative estimate: with 4 replicas each handling 150 messages/s and a target of 100 per pod, the HPA computes ceil(4 × 150 / 100) = 6. If the ratio is close to 1.0, within a configurable tolerance that defaults to about 10%, the HPA skips scaling to avoid needless churn.

The metric type decides what "currentMetric" means:

How it works, step by step

One scaling decision follows this sequence:

1. The signal is produced. Pods export a metric that Prometheus scrapes, or work accumulates in an external system such as a broker.
2. The signal is translated. The Prometheus Adapter turns a metrics API request into a PromQL query. Alternatively, the KEDA operator polls the trigger source on its pollingInterval (30 seconds by default), and the KEDA metrics server returns the value through the external metrics API.
3. KEDA handles zero. If the workload is at zero replicas and the trigger crosses its activation threshold, the KEDA operator scales the workload to one directly. When the trigger has been inactive for the cooldownPeriod (300 seconds by default), KEDA scales it back to zero.
4. The HPA computes a target. On each sync period (15 seconds by default), the HPA controller reads the metric and applies the formula above. If you list several metrics, it takes the highest result.
5. Limits and behavior are applied. The result is clamped to minReplicas and maxReplicas, and then shaped by the behavior stabilization windows and rate policies.
6. The scale subresource is updated. The workload controller creates or removes pods, and the next loop observes the new per-pod value.

The sections below cover each layer in detail: the metrics APIs, the adapter rules, the KEDA objects, and the behavior tuning.

How the metrics APIs, Prometheus Adapter and KEDA fit together

The HPA controller never queries Prometheus or a message broker directly. It only reads from three aggregated Kubernetes APIs, and each metric type maps to one of them:

• metrics.k8s.io serves Resource metrics (CPU, memory). It is normally backed by metrics-server, which scrapes the kubelets.
• custom.metrics.k8s.io serves Pods and Object metrics, which are values bound to Kubernetes objects. The Prometheus Adapter translates these API requests into PromQL queries using its configured rules.
• external.metrics.k8s.io serves External metrics, such as queue depth, stream lag or a cloud provider's API.

Each API is registered through an APIService object, and an APIService points to exactly one backing service. That makes registration exclusive. Only one adapter can own external.metrics.k8s.io at a time. KEDA's metrics server claims that API, so it conflicts with any other external metrics adapter, including a Prometheus Adapter configured to serve external metrics. KEDA can still read from Prometheus through its own Prometheus scaler.

The full path looks like this:

1. Workloads expose metrics, or a broker holds the queue.
2. Prometheus scrapes the workload metrics. KEDA polls the broker.
3. The adapter or the KEDA metrics server answers queries on the metrics APIs.
4. HPAs read those APIs. With KEDA, the operator generates the HPAs from your ScaledObjects.
5. The HPAs scale the Deployment.

Every hop in that chain is now a scaling dependency. If Prometheus, the adapter or the KEDA metrics server is down, the HPA cannot fetch the metric and keeps the current replica count. If the source is stale, the HPA keeps scaling on old values. Neither failure scales you down, but neither scales you up during a spike, so monitor the pipeline as you would any other production component.

Copy+-----------------------+  +----------------+  +-----------------------+
|  Workloads /metrics   |  |    kubelets    |  | Broker, queue, cloud  |
+-----------------------+  +----------------+  +-----------------------+
            v scrape                v scrape               v poll
+-----------------------+  +----------------+  +-----------------------+
|      Prometheus       |  | metrics-server |  | KEDA operator +       |
|                       |  |                |  | KEDA metrics server   |
+-----------------------+  +----------------+  +-----------------------+
            v PromQL                |                      |
+-----------------------+           |                      |
|  Prometheus Adapter   |           |                      |
+-----------------------+           |                      |
            v APIService            v APIService           v APIService
+-----------------------+  +----------------+  +-----------------------+
| custom.metrics.k8s.io |  | metrics.k8s.io |  |external.metrics.k8s.io|
|  Pods/Object metrics  |  |    Resource    |  | External, one owner   |
+-----------------------+  +----------------+  +-----------------------+
            |                       |                      |
            +-----------------------+----------------------+
                                    v read
                       +-------------------------+
                       |  HPAs (HPA controller)  |  <-- generated by KEDA
                       +-------------------------+      from ScaledObjects
                                    v scale
                       +-------------------------+
                       |       Deployment        |
                       +-------------------------+

The HPA reads only the three aggregated metrics APIs, each owned by one backing service; if any hop is down or stale, the HPA holds or scales on old values.

Implementing per-pod scaling with the Prometheus Adapter and a plain HPA

Start with a Prometheus series that measures work per pod. For an HTTP service that is usually a request counter such as http_requests_total, or a gauge of in-flight requests. Each adapter rule has four parts, and each one fails in its own way.

• seriesQuery selects the series to expose. Require namespace and pod labels here, for example http_requests_total{namespace!="",pod!=""}.
• resources.overrides maps those labels to the Kubernetes namespace and pod resources. This mapping is how the adapter attaches a value to an individual pod.
• name renames the metric. A common pattern matches ^(.*)_total$ and rewrites it as ${1}_per_second.
• metricsQuery wraps the series, usually sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>). Without rate(), the HPA divides an ever-growing counter by your target and scales to the maximum.

Before writing an HPA, query the custom metrics API directly with kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/<ns>/pods/*/http_requests_per_second". You should see one item per running pod. If you get a 404 or an empty list, fix the rule first.

The following mistakes produce no error:

• Missing override. If the pod override is missing or uses the wrong label, the series is discovered but never attached to pods.
• Wrong name. If the as name differs from the name the HPA uses, the HPA reports FailedGetPodsMetric and holds its current replica count.
• Discovery delay. The adapter re-lists series on an interval, so a new metric shows up late.

In the autoscaling/v2 HPA, use a Pods metric with an AverageValue target, such as 50 requests per second per pod. Pods metrics accept only AverageValue. That target fits load that spreads evenly across replicas. Use Value on an Object metric when a single number describes the whole workload, such as total requests through an Ingress. The HPA then computes replicas as current replicas × (observed ÷ target).

Copyrules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      # Map Prometheus labels to Kubernetes resources. If the pod label
      # is wrong or missing, values are never attached to pods, and no
      # error is reported.
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total$"
    as: "${1}_per_second"   # must match metric.name in the HPA exactly
  # rate() turns the ever-growing counter into requests per second.
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue   # Pods metrics accept only AverageValue
        averageValue: "50"   # aim for about 50 req/s per pod

A Prometheus Adapter rule that turns http_requests_total into a per-pod http_requests_per_second metric, and an autoscaling/v2 HPA that targets an average of 50 requests per second per pod.

KEDA queue scaling with ScaledObject and ScaledJob

A ScaledObject points KEDA at a Deployment (or any resource with a scale subresource) and a trigger, such as a RabbitMQ queue or a Kafka consumer group. KEDA then creates and owns an ordinary HPA, named keda-hpa-<scaledobject-name>, and serves the trigger's value through the external metrics API. The HPA handles scaling from one replica to many. The KEDA operator handles activation between zero and one replica, which a plain HPA cannot do. When the metric rises above the activation threshold, KEDA scales the workload to one. After the cooldown period with no activity, it scales back to zero.

Base the per-replica target on measured throughput, not a round number. Illustrative estimate: suppose one pod processes 40 messages per second and you want any backlog drained within 30 seconds. Then a target of 1,200 messages per replica means a 12,000-message queue produces 10 replicas.

Use a ScaledJob when tasks are long-running and must not be interrupted. The HPA scales a Deployment down by terminating pods, possibly mid-task. A ScaledJob instead creates a Kubernetes Job for pending work, and each Job runs until it completes.

Know the limits:

• Uneven message cost. Queue length counts messages, not work. If message cost varies widely, scaling on age of the oldest message or processing latency tracks real load better.
• Partition count caps parallelism. Kafka consumers in one group cannot outnumber partitions in any useful way. KEDA's Kafka scaler caps replicas at the partition count by default.
• Do not edit the generated HPA. KEDA reconciles the HPA and overwrites manual changes. Set scaling behavior through advanced.horizontalPodAutoscalerConfig in the ScaledObject instead.

CopyapiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker          # KEDA creates and owns HPA "keda-hpa-order-worker"
  namespace: orders
spec:
  scaleTargetRef:
    name: order-worker        # Deployment to scale
  pollingInterval: 15         # seconds between trigger checks
  cooldownPeriod: 300         # idle seconds before KEDA scales 1 -> 0
  minReplicaCount: 0          # allow scale to zero; the operator handles 0 <-> 1
  maxReplicaCount: 30
  advanced:
    # Tune scaling behavior here, not on the generated HPA,
    # because KEDA overwrites manual edits to that HPA.
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 120
          policies:
            - type: Percent
              value: 50
              periodSeconds: 60
  triggers:
    - type: rabbitmq
      metadata:
        queueName: orders
        mode: QueueLength
        # Illustrative estimate: 40 msg/s per pod x 30 s drain target = 1200
        value: "1200"
        # Wake from zero as soon as any message is waiting
        activationValue: "0"
        # AMQP URL read from the Deployment's container env, e.g.
        # amqp://user:placeholder@rabbitmq.orders.svc:5672/
        hostFromEnv: RABBITMQ_HOST

A ScaledObject that scales a RabbitMQ consumer Deployment from zero on queue length, with a per-replica target derived from measured throughput and scale-down behavior set through advanced.horizontalPodAutoscalerConfig instead of on the generated HPA.

Tuning HPA behavior to prevent flapping

Every stage between real load and a new pod adds lag. Prometheus scrapes on an interval, rate() averages over its own window, the adapter or KEDA polls, and the HPA controller reconciles on its sync period (15 seconds by default). A 2-minute rate() window alone means the metric trails a burst by roughly a minute. Teams often respond with large scale-up percentages and no scale-down window. The result is that replicas chase stale data, overshoot, drop when the burst ends, and spike again on the next burst. That oscillation is flapping. The better fix is to shorten the signal path with a smaller rate() window or a direct queue-depth metric, and then let the behavior field damp the remaining noise.

Illustrative estimate. Assume a 1-minute burst every 4 minutes that needs 20 replicas and a 4-replica baseline. With a scale-down window of 0, the HPA drops toward 4 after each burst and must scale up again every cycle, paying cold-start latency each time. With a 300-second window, the HPA holds the peak recommendation between bursts and stays at 20. You trade some idle capacity for stable latency.

Set maxReplicas from downstream limits, such as database connection pool size divided by connections per pod, or the API rate limit divided by per-pod throughput. Do not derive it from cluster capacity. Scale-down is safe only when consumers are idempotent, handle SIGTERM by finishing or requeueing in-flight work, and have a terminationGracePeriodSeconds longer than the longest message they process.

CopyapiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-worker
spec:
  scaleTargetRef:
    name: orders-worker          # Deployment; set terminationGracePeriodSeconds there to cover in-flight work
  minReplicaCount: 4
  # Cap from downstream limits, not cluster size:
  # e.g. 200 DB connections / 10 connections per pod = 20 replicas
  maxReplicaCount: 20
  pollingInterval: 15            # how often KEDA checks the trigger
  triggers:
    - type: prometheus
      metadata:
        serverAddress:
        # Short rate() window keeps the signal close to real load
        query: sum(rate(orders_processed_total[1m]))
        threshold: "50"         # target per-replica throughput
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0   # react to bursts immediately
          selectPolicy: Max               # take the more aggressive policy
          policies:
            - type: Percent
              value: 100                  # at most double every 15s
              periodSeconds: 15
            - type: Pods
              value: 4                    # or add 4 pods, whichever is larger
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300 # hold the peak between bursts
          selectPolicy: Min               # take the more conservative policy
          policies:
            - type: Percent
              value: 10
              periodSeconds: 60
            - type: Pods
              value: 2
              periodSeconds: 60

A KEDA ScaledObject that passes HPA behavior settings through to scale up quickly on bursts, scale down slowly with a 300-second stabilization window, and cap replicas at a limit derived from database connection capacity.

Verifying and debugging autoscaling

Start with kubectl describe hpa <name> and read the conditions before the events. AbleToScale tells you whether the controller can change the target's scale. ScalingActive set to False means it could not compute a replica count at all, usually because a metric fetch failed. ScalingLimited set to True means the math worked but minReplicas, maxReplicas or a behavior policy clamped the result. The events section names the failing call, such as FailedGetPodsMetric or FailedGetExternalMetric, and often includes the error returned by the API server.

When kubectl get hpa shows <unknown> for a target, query the metrics API yourself to get past the HPA. Use kubectl get --raw against /apis/custom.metrics.k8s.io/v1beta1 or /apis/external.metrics.k8s.io/v1beta1, with the namespace and metric name in the path. An empty list or a 404 means the problem is upstream of the HPA. Then run kubectl get apiservice to confirm which service backs each API group and whether it reports Available. Only one backend can serve a given group.

For KEDA, kubectl get scaledobject shows READY and ACTIVE. KEDA also generates an HPA named keda-hpa-<scaledobject-name>, so the workflow above applies to it too. Scaler errors, such as authentication failures or an unreachable broker, appear in the KEDA operator logs and not in HPA events.

Alternatives to custom-metric HPAs and KEDA

Custom-metric scaling is one option among several. Depending on the workload, one of these may fit better or work alongside it:

• CPU/memory HPA. No extra components are needed beyond metrics-server. It works well when load really is CPU-bound and evenly spread across pods.
• Vertical Pod Autoscaler (VPA). The VPA adjusts the resource requests of each pod instead of the replica count. It suits workloads that cannot run as multiple replicas, or whose requests are poorly sized. Do not let the VPA and an HPA act on the same CPU or memory metric for the same workload, because the two controllers will fight each other.
• Scheduled scaling. If load follows a predictable timetable, you can change replica counts on a schedule. Options include a CronJob that runs kubectl scale, or KEDA's cron trigger. This is simpler than reacting to metrics when the pattern is reliable.
• Request-driven serverless platforms. Knative Serving autoscales on concurrency or requests per second and can scale to zero. It buffers requests while pods start, which addresses the synchronous-HTTP limitation described below. The trade-off is that you adopt its whole serving model, not just an autoscaler.
• Static over-provisioning. You set a fixed replica count sized for peak load. It costs more in idle capacity, but it has no scaling pipeline that can fail, and for small or critical services that can be the right trade.
• A custom controller. You can write your own operator that sets replicas from domain logic. This gives full control, but you then own the code, the testing and the failure modes that KEDA and the HPA already handle.

The Cluster Autoscaler, or another node autoscaler, is not an alternative. It adds nodes when pods are unschedulable, and it complements any of the pod-level approaches above.

How custom-metric scaling saves engineering time

The time savings come from removing recurring manual work and a class of incidents. Speed of scaling is a smaller factor.

• No hand-tuned replica counts. Without autoscaling on the real signal, someone periodically revisits replica counts as traffic grows or shifts. A target per replica, derived once from measured throughput, replaces that recurring review.
• Fewer backlog incidents. If a CPU-scaled consumer sits idle while its queue grows, that failure surfaces as a page and a manual kubectl scale. Scaling on queue length or lag handles the same situation without anyone stepping in.
• No custom glue code. KEDA's built-in scalers cover common brokers, streams and Prometheus queries. Teams do not have to write and maintain their own pollers or metrics adapters for each event source.
• Scale-to-zero without scripts. Idle workers in development, staging or batch environments shut down and wake up automatically. You do not need scheduled jobs to start and stop them.
• One reviewable, repeatable pattern. ScaledObjects and HPAs are declarative manifests. They live in version control, go through code review, and can be copied across services. Each new workload reuses the same pattern instead of a bespoke scaling approach.
• Faster diagnosis. The conditions, events and raw metrics API queries described above give one standard debugging path. That replaces ad hoc investigation for every service.

These savings are offset by the operational cost of running the extra components. The next section shows how to weigh the two.

Choosing an approach: when to use each, when to avoid them, and cost impact

Start with the simplest signal that tracks real work, and add machinery only when that signal fails.

Skip custom-metric scaling in three cases:

• Synchronous HTTP traffic. A request cannot wait for a pod to schedule and pass readiness. Put a buffering proxy in front, or keep minReplicas at 1.
• No reliable metrics pipeline. If the HPA cannot fetch the metric, it keeps the current replica count. The failure is quiet, so you may not notice it during a traffic spike.
• Downstream limits. If a database or a rate-limited API already caps throughput, extra replicas only add contention. Scale the bottleneck or lower maxReplicas.

The cost is real. You take on CRDs and an operator to upgrade, adapter rules to maintain, and another component on the on-call path. Compare that against the time you no longer spend hand-tuning capacity and the incidents caused by scaling on CPU while a queue backs up.

Illustrative estimate (assumed numbers, not a benchmark): net hours/month = manual tuning + (incidents × hours per incident) − maintenance

With 4 h of tuning, 2 incidents at 3 h each, and 3 h of maintenance: 4 + (2 × 3) − 3 = 7 hours saved per month.

If the result is near zero or negative, stay on the CPU-based HPA.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to hpa custom metrics keda, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.