Kubernetes HPA Custom Metrics: Prometheus Adapter vs KEDA

How the HPA gets metrics beyond CPU

The HPA controller does not collect metrics. On each sync loop it queries the Kubernetes API for the current value of every metric in its spec, computes a desired replica count, and updates the target's scale subresource. Which API it queries depends on the metric type.

CPU and memory come from metrics.k8s.io, the resource metrics API. metrics-server implements it by reading usage from each kubelet. It only knows about container resource usage, so it cannot report queue depth, request rate, or anything else your application measures.

For other metrics, the HPA queries two more APIs: custom.metrics.k8s.io for metrics attached to Kubernetes objects such as Pods or Ingresses, and external.metrics.k8s.io for metrics that have no in-cluster object, such as a cloud queue's backlog. Kubernetes defines these APIs but does not implement them. A component must register an APIService with the aggregation layer and serve the responses. Until one does, the HPA reports that it cannot fetch the metric and does not scale on it.

The full pipeline looks like this:

1. Your application exposes a metric through an exporter or client library.
2. Prometheus scrapes it and stores the time series.
3. Prometheus Adapter or KEDA queries Prometheus and translates the result into the metrics API format.
4. The API server routes the HPA's request to that component through the aggregated API.
5. The HPA controller calculates replicas and scales the Deployment.

Each aggregated API group is backed by a single APIService registration, so only one component can serve external.metrics.k8s.io in a cluster. Prometheus Adapter and KEDA cannot both provide external metrics, so you have to pick one for that role.

Copy +-----------------+   scrape   +-------------+
 | App + exporter  |----------->| Prometheus  |
 +-----------------+            +-------------+
                                       ^
                                       | PromQL query
 +-----------------+     +----------------------------+
 | metrics-server  |     | Prometheus Adapter         |
 | (reads kubelet) |     |   OR KEDA                  |
 +-----------------+     | only one may serve         |
          ^              | external.metrics.k8s.io    |
          |              +----------------------------+
          |                            ^
          | metrics.k8s.io             | custom.metrics.k8s.io
          |                            | external.metrics.k8s.io
 +-------------------------------------------------------+
 |          kube-apiserver (aggregation layer)           |
 +-------------------------------------------------------+
                             ^
                             | HPA queries metric values each sync loop
                   +--------------------+
                   |   HPA controller   |
                   +--------------------+
                             |
                             | update scale subresource
                             v
                   +--------------------+
                   |     Deployment     |
                   +--------------------+

The HPA reads CPU and memory from metrics-server via metrics.k8s.io, and reads other metrics via the custom and external metrics APIs, which Prometheus Adapter or KEDA serves from Prometheus data through the aggregation layer.

How it works, step by step

The diagram shows the components. The control loop that runs through them works like this:

1. Sync. The HPA controller runs in kube-controller-manager and evaluates every HPA on a fixed sync period. The default is 15 seconds, set by the --horizontal-pod-autoscaler-sync-period flag.
2. Fetch. For each entry in spec.metrics, the controller calls the matching metrics API through the API server. The aggregation layer forwards custom and external metric requests to whichever component registered that APIService.
3. Translate. With Prometheus Adapter, the adapter fills in the matching rule's metricsQuery template, runs the PromQL against Prometheus, and returns the result as a metric value per object. With KEDA, KEDA's metrics server runs the trigger's query or calls the event source through its scaler and returns an external metric value.
4. Compute. The controller applies desiredReplicas = ceil(currentReplicas * currentValue / targetValue) for each metric, skips any change inside the tolerance band, and takes the highest result across metrics.
5. Shape. The behavior policies and stabilization windows limit how far and how fast the count may move. minReplicas and maxReplicas clamp the result.
6. Act. The controller writes the new replica count to the target's scale subresource. The Deployment controller then creates or removes pods.

KEDA adds a loop of its own on top of this. Its operator polls each trigger every pollingInterval seconds to decide whether the workload should be active at all. That loop handles the move between zero and one replica. Between one replica and the maximum, the HPA that KEDA generated does the scaling through the same steps as above.

Who should use it

Custom-metric autoscaling fits teams that already run Prometheus, or that consume events from a broker, and whose services see load that CPU does not reflect. Typical owners:

• Backend engineers running request-driven services such as HTTP or gRPC APIs. Request rate or in-flight requests per pod track demand more closely than CPU for I/O-bound handlers.
• Engineers running queue or stream consumers. Backlog size is the direct measure of work waiting, and it is the signal KEDA's scalers are built around.
• Platform teams that provide autoscaling as a shared cluster capability. Someone has to own the adapter or KEDA installation, the APIService registration, and the Prometheus rules that other teams depend on.

If nobody can take ownership of the metrics pipeline, a CPU-based HPA with conservative bounds is the safer default. A broken custom metric stops scaling.

When to use it

Use custom or external metrics when:

• CPU stays flat or low while latency or backlog grows, because work is blocked on I/O, locks, or downstream calls that pods can share.
• You can name a metric that is proportional to per-pod load and drops as replicas are added, as described in the next section.
• Load arrives from outside the cluster, for example a message queue, and the backlog is visible through Prometheus or a KEDA scaler.
• A workload sits idle for long stretches and can absorb a cold start, so scaling to zero with KEDA saves the cost of idle replicas.
• Load changes often enough that hand-tuned replica counts either waste capacity or cause pages.

When to avoid it

Avoid custom-metric autoscaling, or stick with simpler options, when:

• The bottleneck is downstream. If a shared database, a rate-limited third-party API, or a fixed-size connection pool limits throughput, more pods add load to that bottleneck without increasing throughput.
• The only available signals are outcome metrics such as latency or error rate. These do not fall as replicas are added, so the HPA scales toward maxReplicas.
• The workload cannot use extra replicas. Examples include a consumer group with fewer partitions than the replicas you would add, a singleton that relies on leader election, or a stateful service that does not rebalance work across new pods.
• Pod startup is slow compared with how quickly traffic changes. The HPA reacts after load arrives. If a new pod takes minutes to become ready, provisioning ahead of time or scheduled scaling works better.
• The cluster has no spare capacity and no node autoscaling. A higher replica count only produces Pending pods.
• CPU already tracks load. For CPU-bound services with uniform requests, a CPU-only HPA gives the same result with fewer moving parts.
• The workload is a run-to-completion batch job. A Deployment that scales to zero still treats work as long-running pods. Kubernetes Jobs, or KEDA's ScaledJob resource, model this better.

Choosing a metric that tracks per-pod load

CPU is a weak proxy for request-driven or queue-driven services. A handler blocked on I/O can sit at low CPU while its queue grows, and the HPA sees nothing to act on. Backend and platform engineers running APIs, gRPC services, or queue consumers get faster and more predictable scaling from a metric that measures demand directly.

The test for a good metric is simple: it must be proportional to per-pod load and fall as replicas are added. Requests per second per pod, in-flight requests per pod, and queue depth per worker all pass, because new pods take a share of the work and the per-pod value drops.

Outcome metrics fail this test. If p99 latency or error rate rises because a downstream database is saturated, more pods will not lower it. The HPA keeps scaling toward maxReplicas, and the extra connections make the database bottleneck worse.

The controller computes desiredReplicas = ceil(currentReplicas * currentValue / targetValue). As an illustrative estimate, assume 4 pods, an observed 150 rps per pod, and a target of 100 rps: ceil(4 * 150 / 100) = 6 replicas. If the ratio is close to 1.0, within a default tolerance of about 10%, the controller does nothing, which prevents flapping.

Exposing a custom metric with Prometheus Adapter

Prometheus Adapter turns PromQL into the custom metrics API through rules, and each rule answers four questions. seriesQuery selects the raw series to discover, such as http_requests_total{namespace!="",pod!=""}. resources maps Prometheus labels to Kubernetes objects: an overrides entry ties the namespace label to the namespace resource and pod to the pod resource. Without that mapping the adapter cannot answer "value for pod X." name renames the series, typically turning a _total counter into http_requests_per_second. metricsQuery is the template the adapter actually runs. It uses <<.Series>>, <<.LabelMatchers>> and <<.GroupBy>> so one rule can serve any namespace or pod set the HPA asks for.

For counters, wrap the series in rate() with a window of at least four scrape intervals. A shorter window can fall between samples and return gaps. A longer one delays scale-up by smoothing spikes the HPA should react to.

Several mistakes produce numbers that look reasonable but describe the wrong scope:

• Hardcoded aggregation. Using sum(...) by (namespace) instead of by (<<.GroupBy>>) returns the namespace total for every pod.
• Scraping through a Service or sidecar exporter. The pod label then names the exporter's pod, not the workload's pod.
• Relabeling mismatches. Relabeling that renames pod to kubernetes_pod_name silently breaks the override.

Before writing any HPA, query the API directly with kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/<ns>/pods/*/http_requests_per_second". Confirm that each listed pod belongs to the target workload and that the values add up to what Prometheus reports.

Copyrules:
  # 1. Discovery: only series that carry both labels can be mapped to a pod.
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    # 2. Association: tie Prometheus labels to Kubernetes resources.
    #    If relabeling renamed `pod` (e.g. to kubernetes_pod_name), this
    #    override must name that label instead, or lookups fail.
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    # 3. Naming: expose the counter as a per-second rate metric.
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    # 4. Query template the adapter fills in for each API request.
    #    Group by <<.GroupBy>> so each pod gets its own value.
    #    A hardcoded `by (namespace)` here would return the namespace
    #    total for every pod.
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

A Prometheus Adapter rule that discovers a request counter, maps its namespace and pod labels to Kubernetes resources, renames it to a per-second metric, and aggregates by the requested group so values stay per pod, followed by the kubectl command that checks the result.

Writing an autoscaling/v2 HPA with Pods, Object, and External metrics

Once the adapter returns sane numbers, the HPA can consume them through three metric types. Each one answers a different question.

• Pods averages a metric across the target's pods and compares it to an AverageValue, such as 50 requests per second per pod. Use it when every pod reports its own load and that load divides evenly as replicas are added.
• Object reads a single value from another Kubernetes object, named in describedObject. A common case is requests per second on an Ingress. It fits signals that belong to a shared resource rather than to individual pods.
• External reads a metric with no Kubernetes object behind it, such as a queue depth from a cloud broker. With an AverageValue target, the HPA divides the total by the replica count, so "messages per pod" maps directly to a target.

If you list several metrics, the HPA computes a desired replica count for each one and uses the highest. Adding a metric therefore can only push scaling up. It never caps it.

The behavior field controls how quickly replicas change:

• Policies limit change per periodSeconds, either by pod count or by percentage.
• selectPolicy chooses the more or less aggressive of those policies.
• stabilizationWindowSeconds makes the HPA act on the most conservative recommendation within the window.

Scale-down has a default window of 300 seconds because load metrics are noisy. Without the window, a brief dip removes pods that are needed again a minute later. Each removal also drops connections and warm caches.

Shortening the window cuts idle capacity, at the cost of flapping and connection churn. Lengthening it gives stability, and you pay for replicas that sit idle.

CopyapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: web}
  minReplicas: 3
  maxReplicas: 30
  # The HPA computes a replica count for each metric and uses the highest.
  metrics:
  - type: Pods            # per-pod load, averaged across pods
    pods:
      metric: {name: http_requests_per_second}
      target: {type: AverageValue, averageValue: "50"}
  - type: Object          # one value read from a shared Ingress
    object:
      metric: {name: requests_per_second}
      describedObject: {apiVersion: networking.k8s.io/v1, kind: Ingress, name: web}
      target: {type: Value, value: "2k"}
  - type: External        # no Kubernetes object behind it, e.g. broker queue depth
    external:
      metric:
        name: queue_messages_ready
        selector: {matchLabels: {queue: orders}}
      target: {type: AverageValue, averageValue: "30"}  # total / replicas = msgs per pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0   # react to spikes immediately
      selectPolicy: Max               # allow the larger of the two policies
      policies:
      - {type: Percent, value: 100, periodSeconds: 15}
      - {type: Pods, value: 4, periodSeconds: 15}
    scaleDown:
      stabilizationWindowSeconds: 300 # default; ignore brief dips in noisy metrics
      selectPolicy: Min               # remove pods at the slower of the two rates
      policies:
      - {type: Percent, value: 20, periodSeconds: 60}
      - {type: Pods, value: 2, periodSeconds: 60}

An autoscaling/v2 HPA that combines Pods, Object, and External metrics, where the highest per-metric replica count wins, with a behavior block that scales up quickly and scales down slowly behind a 300-second stabilization window.

Using KEDA for event sources and scale to zero

KEDA starts from a different premise than Prometheus Adapter. You do not write the HPA yourself. You create a ScaledObject that names a target Deployment, a minimum and maximum replica count, and one or more triggers. KEDA then generates and owns a standard HPA for that workload. Each trigger uses a scaler that knows how to read a specific source, such as a Kafka consumer group's lag, a RabbitMQ or SQS queue depth, or a PromQL query. The example below uses a Prometheus trigger. It sets minReplicaCount to 0, a query returning queued jobs, a threshold for the per-replica target, and an activationThreshold that decides when to wake from zero.

That activation step is the key difference. Without non-default feature gates, the HPA cannot move a workload between zero and one replica. KEDA's operator handles that transition itself and hands control to the HPA once at least one pod exists.

The Kubernetes API aggregation layer allows only one backend per metrics API. If both tools run in one cluster, let Prometheus Adapter serve the custom metrics API and leave the external metrics API to KEDA.

Scale to zero has a cost. The first message or request after an idle period waits for scheduling, image pull, container start, and readiness before any work happens. Use it for queue consumers that can tolerate that delay, not for latency-sensitive request paths.

CopyapiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: job-worker
  namespace: workers
spec:
  scaleTargetRef:
    name: job-worker          # Deployment to scale; KEDA creates and owns its HPA
  minReplicaCount: 0          # allow scale to zero when the queue is idle
  maxReplicaCount: 20
  pollingInterval: 30         # seconds between trigger checks
  cooldownPeriod: 300         # wait 5 minutes of inactivity before dropping to zero
  triggers:
    - type: prometheus
      metadata:
        serverAddress:
        # Total jobs waiting in the queue across all producers
        query: sum(jobs_queued{queue="reports"})
        # Per-replica target: the HPA aims for ~50 queued jobs per pod
        threshold: "50"
        # Wake from zero only once more than 5 jobs are queued
        activationThreshold: "5"

A KEDA ScaledObject using a Prometheus trigger, where threshold sets the per-replica target for the generated HPA and activationThreshold controls when KEDA moves the Deployment from zero to one replica.

Debugging an HPA that will not scale

Start with kubectl describe hpa <name>. The Metrics line shows each target as current versus desired. A value of <unknown> means the controller could not fetch that metric. The Conditions block tells you which stage of the control loop failed:

• AbleToScale reports whether the controller can read and update the target's scale subresource. If it is False, the HPA cannot change replicas, so check that the target reference is correct.
• ScalingActive reports whether a replica count could be computed. False here almost always points to a metrics problem. The Reason and Message fields usually name the API call that failed.
• ScalingLimited means the desired count was clamped by minReplicas, maxReplicas, or a scaling policy. The HPA is working, but your bounds are stopping it.

Read the Events below the conditions. Messages such as FailedGetPodsMetric or FailedGetExternalMetric carry the underlying error.

Next, query the metrics APIs yourself with kubectl get --raw against the custom.metrics.k8s.io or external.metrics.k8s.io paths. The HPA calls these same endpoints, so this shows you exactly what it sees. Run kubectl get apiservices to confirm the API registration is Available. If it is not, read the Prometheus Adapter logs, or the logs of KEDA's operator and metrics server. Common causes are a PromQL query that returns no series and label mismatches between the series and the pod or namespace.

When a metrics source is down, the HPA cannot compute that metric. With several metrics configured, it may still scale up based on the ones that respond, but it will not scale down until every metric is readable again.

If replicas rise but the new pods stay Pending, the HPA has done its job. The scheduler has no node with room for them, so you need a cluster autoscaler or more capacity.

Alternatives

Custom-metric HPAs are one option among several. Some of the options below replace it, and others work alongside it.

• CPU or memory HPA with metrics-server. This needs no Prometheus dependency and has fewer failure points. It works well when CPU tracks load, and it can be combined with custom metrics in the same HPA because the highest replica count wins.
• Fixed replicas with headroom. For steady, predictable traffic, a static replica count sized for peak load is simple and has no scaling lag. You pay for idle capacity during off-peak hours.
• Scheduled scaling. If load follows a known daily or weekly pattern, you can change replica counts or minReplicas on a schedule. Options include a CronJob that patches the resource, or KEDA's cron scaler. This adds capacity before the load arrives, which reactive scaling cannot do.
• Vertical Pod Autoscaler (VPA). VPA changes resource requests per pod instead of the pod count. It fits workloads that cannot scale horizontally or whose requests are badly sized. Do not let VPA and an HPA both act on CPU or memory for the same workload, because they will work against each other.
• Cluster or node autoscaling. This is a complement, not a replacement. It adds nodes when pods are Pending, so HPA-driven scale-up has somewhere to land.
• Application-level concurrency control. Bounded worker pools, backpressure, and load shedding keep a fixed set of pods healthy under bursts. They pair well with autoscaling because they protect the service during the scaling lag.
• Kubernetes Jobs or KEDA ScaledJob for batch work. When each unit of work runs to completion, creating a Job per batch avoids the need to scale a long-running Deployment and drain it afterwards.

Between the two tools in this article, choose Prometheus Adapter when every signal already lives in Prometheus and you want to write plain HPAs. Choose KEDA when you need scale to zero or want to read event sources directly through its scalers.

Impact on cost, latency, and engineering time

Every stage between a request arriving and a new pod serving traffic adds delay, so the HPA always reacts to load that has already arrived. The approximate lag is:

lag ≈ scrape interval + rate window + adapter cache + HPA sync period + pod startup

Illustrative estimate (assumed numbers, not a benchmark): 30 s + 60 s + 30 s + 15 s + 20 s ≈ 155 s. If traffic can double within that time, the running pods must absorb the spike on their own. You can set the target below saturation (for example 60% of what a pod can handle) or keep a minimum replica count that covers the burst. Either choice means paying for idle capacity to buy time.

Custom metrics also add a dependency. If Prometheus, the adapter, or KEDA's event source fails, scaling stops. The HPA still relies on accurate resource requests and on free cluster capacity, because a correct replica count does not help when the new pods sit in Pending.

Scale to zero removes the cost of idle replicas. In exchange, the first request after an idle period waits for a cold start, so it suits queue consumers better than synchronous APIs. Engineers spend less time hand-tuning replica counts and get fewer capacity pages, but someone has to own the metrics pipeline.

Decision checklist

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes hpa custom metrics, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.