Kubernetes HPA Custom Metrics: Scaling on Real Demand Signals

Why CPU is a weak scaling signal and what to scale on instead

The Horizontal Pod Autoscaler adjusts replica counts to keep a metric near a target. By default that metric is CPU or memory utilization. Custom metrics scaling lets the HPA act on a signal from your application, such as requests per second or messages waiting in a queue, so replica count follows actual demand.

CPU is an indirect signal. An I/O-bound API that spends most of its time waiting on a database can have a growing backlog while CPU stays low. A queue consumer that blocks on network calls shows the same pattern: the queue grows and CPU barely moves. Memory is often worse, because many runtimes do not release heap after a spike, so usage stays high after load has dropped.

A good scaling metric has two properties. It rises in proportion to load, and it can be divided per replica, so that adding pods lowers the value each pod sees.

These metrics help most when traffic is bursty or event-driven. CPU-based scaling is still good enough for compute-bound services, where CPU tracks the work being done.

Be careful about scaling on latency. Latency often rises because a downstream dependency is slow, not because your service lacks capacity. If the autoscaler adds replicas in that situation, they send more concurrent requests to the struggling database or service. This can make the slowdown worse, and the slower responses then trigger more scaling.

How the HPA reads metrics: resource, custom and external metrics APIs

The HPA controller runs inside kube-controller-manager and evaluates every HPA object on a fixed loop, every 15 seconds by default. It never talks to Prometheus or your application directly. It queries three aggregated APIs served through the Kubernetes API server:

• metrics.k8s.io provides CPU and memory for resource metrics, usually from metrics-server.
• custom.metrics.k8s.io provides metrics attached to Kubernetes objects, such as requests per second per pod.
• external.metrics.k8s.io provides metrics with no Kubernetes object behind them, such as queue depth in a broker.

Each group is registered with an APIService object, which tells the API server which in-cluster service handles requests for that group. Because one APIService maps one group to one backend, only one adapter can own external.metrics.k8s.io at a time. If you install Prometheus Adapter and KEDA together, decide which one serves external metrics.

To see what your cluster exposes, run kubectl get apiservices and check the AVAILABLE column for the metrics groups. Then query a group directly, for example kubectl get --raw /apis/custom.metrics.k8s.io/. An empty or failing response here explains most "unknown" metric errors in kubectl describe hpa.

The pipeline and its delays

1. App to Prometheus: delay is up to one scrape interval.
2. Prometheus to adapter or KEDA: delay includes query evaluation plus any rate window, so a rate() over a 2-minute window smooths out short spikes.
3. Adapter to API server: delay includes adapter caching or polling.
4. API server to HPA controller: delay is up to one sync period.
5. HPA to Deployment: delay covers pod scheduling, image pull, and readiness.

These delays add up. Expect tens of seconds to a few minutes between a change in load and a ready new replica.

Copy#!/usr/bin/env bash
# Inspect the metrics pipeline the HPA depends on, from registration to served values.
set -euo pipefail
NS=shop   # placeholder namespace

# 1. Which aggregated metrics APIs are registered, and are they healthy?
#    AVAILABLE=False here means the HPA will report metrics as <unknown>.
kubectl get apiservices | grep -E 'NAME|metrics.k8s.io'

# 2. Which backend Service owns each group? Only one can own external metrics.
for svc in v1beta1.metrics.k8s.io v1beta1.custom.metrics.k8s.io v1beta1.external.metrics.k8s.io; do
  owner=$(kubectl get apiservice "$svc" \
    -o jsonpath='{.spec.service.namespace}/{.spec.service.name}' 2>/dev/null || echo 'not registered')
  echo "$svc -> $owner"
done

# 3. Resource metrics (CPU/memory), normally served by metrics-server.
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/${NS}/pods"

# 4. Discovery: list every custom/external metric name the adapter exposes.
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1

# 5. Fetch the exact values the HPA will read (metric names are placeholders).
#    Quote the path so the shell does not expand the '*' pod selector.
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/${NS}/pods/*/http_requests_per_second"
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/${NS}/queue_depth"

# 6. Compare with what the controller actually saw on its last sync.
kubectl describe hpa -n "$NS"

A shell walkthrough that checks which aggregated metrics APIs are registered and who owns them, queries each API path directly, and compares the results with the HPA's own view.

Metric types, target types and the scaling formula

Load delay is one reason the target value matters: the HPA can only respond as well as the metric and target you give it. In autoscaling/v2, each entry under spec.metrics declares a metric type and a target type.

Utilization applies only to Resource metrics. It expresses CPU or memory as a percentage of the pod's requests. Value compares the raw metric directly. AverageValue divides the metric by the current replica count first, which makes it the natural choice for queue depth when you want a fixed amount of work per pod.

The controller computes desiredReplicas = ceil(currentReplicas × currentMetricValue / desiredMetricValue). It skips scaling while the ratio stays inside a tolerance band, which defaults to 10%. The band keeps the replica count from flapping on noise. As an example, 4 replicas averaging 150 req/s against a target of 80 give a ratio of 1.875. That is outside the band, so the HPA scales to ceil(7.5) = 8 replicas.

Choosing the target

Illustrative estimate, assumed inputs: a load test shows one pod sustains 120 req/s before p99 latency breaches the SLO. Leaving 30% headroom gives a target of about 84 req/s. The headroom covers the scaling delay described above and lets traffic spikes arrive before new pods are ready. Round the target down to 80 req/s.

That 120 req/s figure applies only to the code you tested. New dependencies, serialization changes and cache behaviour all shift per-pod capacity. Re-run the measurement on a regular schedule and after significant releases.

CopyapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 3
  maxReplicas: 30
  # With several metrics, the HPA computes replicas for each and uses the largest.
  metrics:
  # Pods: per-pod rate averaged across pods. 80 = measured 120 req/s minus ~30% headroom, rounded down.
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "80"
  # Object: one metric read from a single Ingress, compared as a raw total.
  - type: Object
    object:
      metric:
        name: requests_per_second
      describedObject:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        name: checkout-ingress
      target:
        type: Value
        value: "2k"
  # External: queue depth from outside the cluster, divided by replicas for a fixed backlog per pod.
  - type: External
    external:
      metric:
        name: queue_messages_ready
        selector:
          matchLabels:
            queue: orders
      target:
        type: AverageValue
        averageValue: "30"

An autoscaling/v2 HPA with a Pods metric using AverageValue, an Object metric using Value and an External queue-depth metric using AverageValue. The Pods target of 80 comes from the illustrative capacity estimate in this section.

Implementing it with Prometheus Adapter

Prometheus Adapter runs as an aggregated API server that answers custom.metrics.k8s.io requests by translating them into PromQL. Its configuration is a list of rules. Each rule has four parts that work together.

Write the rule

• Discovery (seriesQuery) selects which series the adapter considers, for example http_requests_total{namespace!="",pod!=""}. Requiring non-empty namespace and pod labels filters out series that could never be attached to a pod.
• Resource mapping (resources.overrides) tells the adapter which Prometheus label identifies which Kubernetes resource, such as namespace → namespace and pod → pod. Without this mapping the adapter cannot answer a per-pod query.
• Naming (name.matches / name.as) renames the series. Raw counters should not be used as targets, so rename http_requests_total to http_requests_per_second.
• Query (metricsQuery) converts the counter into a per-pod rate: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>). The adapter fills in the pod and namespace selectors from the HPA request.

Verify before creating the HPA

Run kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/<ns>/pods/*/http_requests_per_second" and confirm that every pod returns a value. If the metric is missing here, the HPA will only report FailedGetPodsMetric. Once values appear, create a Pods-type metric with an AverageValue target set from your measured per-pod capacity.

Common mistakes

• Dropping by (<<.GroupBy>>), which collapses all pods into one number.
• Setting a rate window that covers only a few scrapes, which produces noisy values.
• Writing overly broad discovery queries such as {__name__=~".+"}. The adapter re-runs discovery periodically, so a high-cardinality selector creates recurring expensive load on Prometheus.

Copy# prometheus-adapter config: one rule that exposes a per-pod request rate
rules:
  # Discovery: only series that can be attached to a pod
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    # Resource mapping: Prometheus label -> Kubernetes resource
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    # Naming: expose the rate under a new name, not the raw counter
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    # Query: per-pod rate; keep by (<<.GroupBy>>) or all pods collapse into one value
    # 2m window should span several scrape intervals to avoid noisy values
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
---
# Create only after kubectl get --raw returns a value for every pod
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"   # placeholder: set from measured per-pod capacity

A Prometheus Adapter rule that discovers, maps, renames and converts http_requests_total into a per-pod rate, followed by the Pods-type HPA that targets the resulting http_requests_per_second metric.

Implementing it with KEDA, including scale to zero

KEDA replaces adapter configuration with a custom resource called a ScaledObject. It names a workload through scaleTargetRef, sets minReplicaCount and maxReplicaCount, and lists one or more triggers. The KEDA operator watches ScaledObjects and creates a matching HPA that it owns, typically named keda-hpa-<scaledobject-name>. KEDA's metrics server then serves trigger values through the external metrics API. Edit the ScaledObject, not the generated HPA, because the operator reconciles the HPA back to the ScaledObject's spec.

A Prometheus trigger takes a server address, a PromQL query and a threshold. The HPA treats the threshold as an AverageValue target, so a query returning total request rate divided by a threshold of 100 aims for roughly 100 requests per second per pod. A queue trigger, such as RabbitMQ in QueueLength mode, works the same way with messages per replica as the unit.

Setting minReplicaCount: 0 splits responsibility between two components. The HPA handles scaling between 1 and N replicas. The KEDA operator handles the step from 0 to 1, which it checks every pollingInterval, using the trigger's activation threshold. The reverse step from 1 to 0 happens after cooldownPeriod. This only works if the signal exists while no pods are running. Queue depth or ingress-level request counts qualify. Metrics scraped from the application pods do not, because with zero pods there is nothing to scrape, and the workload stays at zero.

Scaling from zero adds cold-start latency. A request has to wait for the polling interval, scheduling, the image pull, application startup and readiness checks before it is served. That delay is acceptable for queue consumers but often unacceptable for synchronous APIs.

Only one APIService can register v1beta1.external.metrics.k8s.io. If Prometheus Adapter also serves external metrics, the two components conflict over that registration, and HPAs belonging to one of them stop working. To run both, serve only custom metrics from the adapter and leave the external metrics API to KEDA.

Copy# Queue consumer: KEDA owns the HPA (keda-hpa-orders-worker); edit this, not the HPA.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-worker
  namespace: shop
spec:
  scaleTargetRef:
    name: orders-worker        # Deployment to scale
  minReplicaCount: 0           # operator handles 0 <-> 1, HPA handles 1 <-> N
  maxReplicaCount: 20
  pollingInterval: 15          # seconds between activation checks while at zero
  cooldownPeriod: 300          # seconds of inactivity before dropping 1 -> 0
  triggers:
    - type: rabbitmq
      metadata:
        mode: QueueLength
        value: "20"            # target ~20 messages per replica
        activationValue: "0"   # any message wakes the workload from zero
        queueName: orders
        hostFromEnv: RABBITMQ_HOST   # AMQP URL read from the target's container env
---
# Synchronous API: query ingress metrics, which still exist when the API has zero pods.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-api
  namespace: shop
spec:
  scaleTargetRef:
    name: orders-api
  minReplicaCount: 1           # keep one warm pod to avoid cold-start latency
  maxReplicaCount: 30
  triggers:
    - type: prometheus
      metadata:
        serverAddress:
        query: sum(rate(nginx_ingress_controller_requests{exported_service="orders-api"}[2m]))
        threshold: "100"         # AverageValue target: ~100 req/s per pod

Two ScaledObjects: a RabbitMQ QueueLength trigger that scales a consumer to and from zero, and a Prometheus trigger on ingress-level request rate for an API that keeps one warm replica.

Tuning behavior, combining metrics and debugging the autoscaler

Stopping flapping with behavior

Noisy metrics such as queue depth or request rate make the HPA oscillate unless you shape its response. The behavior field has separate scaleUp and scaleDown blocks. Each takes a stabilizationWindowSeconds value and a list of policies. A policy caps change per periodSeconds, either as a number of Pods or as a Percent of current replicas. During the window, the HPA acts on the most conservative recommendation it has seen. Scale-down defaults to a 300-second window, so a brief dip does not remove capacity you will need again in a minute. A common pattern is to scale up immediately, allowing the larger of a doubling or a fixed pod count, and to scale down slowly, a small percentage per minute. Burst traffic gets capacity fast, and one quiet scrape does not drain the fleet.

Combining metrics

When you list several metrics, the HPA computes a replica count for each and uses the highest. Adding a metric can only make scaling more aggressive. Set resource requests anyway. Utilization targets are calculated against requests. The scheduler also needs requests to place new pods without overcommitting nodes.

Debugging

Start with kubectl describe hpa. Three conditions matter:

• AbleToScale reports whether the HPA can fetch and update the scale target.
• ScalingActive turns False when metrics cannot be read.
• ScalingLimited means the min or max replica bound clipped the desired count.

Events such as FailedGetExternalMetric name the query that failed. If metrics are missing, or the reported values never change, query the metrics API directly with kubectl get --raw. Then read the Prometheus Adapter or KEDA operator logs for query errors or series that do not match.

When the backend is down, the HPA keeps its current replica count. Guessing in either direction is unsafe, so it holds where it is. If only some of several metrics are unavailable, it may still scale up on the metrics it can read, but it will not scale down.

Choosing an approach: alternatives, cost and engineering time

Skip custom metrics when load is steady, because a fixed replica count is cheaper to run and to reason about. Skip them when monitoring is immature, because if Prometheus is flaky the autoscaler inherits that flakiness. Avoid latency as the only signal: latency rises for reasons replicas can't fix, such as a slow database, and scaling out can make that worse.

Every extra component has to be upgraded alongside the cluster, monitored and debugged at 3 a.m. The trade is worth it when a good signal replaces manual scaling and paging.

Illustrative estimate (assumed numbers, not a benchmark): monthly saving = (idle replicas avoided × cost per replica-hour × hours) − (component maintenance hours × engineer hourly cost)

Assume you avoid 6 idle replicas at $0.05/hour for 730 hours, which is about $219. Assume upkeep is 2 hours at $100/hour, which is $200. That leaves almost no net saving. In that case the justification has to come from fewer incidents, not from infrastructure spend.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes hpa custom metrics, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.