What is high cardinality in Prometheus and why does it drive metrics costs?

In Prometheus, a time series is one metric name combined with one exact set of label key-value pairs. http_requests_total{method="GET", status="200"} and http_requests_total{method="GET", status="500"} are two separate series. Each one has its own entry in the index, its own in-memory chunk, and its own stream of samples. "High cardinality" means a metric, or a whole server, has a large number of these unique combinations.

The series count matters more than the sample count because most of the expensive parts of Prometheus scale per series:

  • Head-block memory: every active series keeps its labels, index postings and an open chunk in RAM.
  • Index size: each persisted block stores label names, values and postings lists for every series it contains.
  • Compaction work: merging blocks means rewriting indexes whose size follows the number of series.
  • Remote-write volume: each series sends its labels along with its samples, and a receiver has to track every series it ingests.
  • Managed backends: many price by active series, so each new label value is a separate billing unit.

Churn makes this worse. A series that exists for only a few minutes, such as one labeled with a pod name from a single deployment, still takes memory and index space. It is only released when the head block is compacted and that series stops appearing. Frequent deploys can therefore keep many dead series resident at the same time as the live ones.

Your bill depends on which dimensions you put in labels, not on how much you instrument. This affects two groups. Backend engineers choose the labels when they write instrumentation. Platform engineers run Prometheus or pay for the managed backend that receives the result.

 Instrumentation: backend engineers choose the label dimensions
 +----------------------------------------------------------+
 | http_requests_total{method, status, pod}                 |
 +----------------------------------------------------------+
         |  each unique label combination = one time series
         v
 +----------------------------------------------------------+
 | {GET,200,pod-a}  {GET,500,pod-a}  {GET,200,pod-b}  ...   |
 | active series = label combinations actually observed     |
 +----------------------------------------------------------+
         |
         v
 +------------------- Prometheus server --------------------+
 | Head block (RAM), per active series:                     |
 | +-----------------+  +-----------------+                 |
 | | labels+postings |  | open chunk      |                 |
 | +-----------------+  +-----------------+                 |
 |         |  compaction: rewrites index per series         |
 |         v                                                |
 | Persisted blocks: index size grows with series count     |
 +----------------------------------------------------------+
         |  remote write: labels + samples for every series
         v
 +----------------------------------------------------------+
 | Remote receiver / managed backend                        |
 | tracks every series; often billed per active series      |
 +----------------------------------------------------------+
 Churn: pod-a (old deploy) --> stays in head until compaction
        pod-b (new deploy) --> resident alongside pod-a

Each distinct label combination becomes its own series, and head memory, index size, compaction, remote-write volume and managed-backend billing all scale with the number of active and churned series.

How does label cardinality multiply series counts?

Prometheus stores one time series for every unique combination of metric name and label values. The series count is therefore the product of each label's distinct values, not their sum. Each new label multiplies the existing total.

The following is an illustrative estimate, not a measurement. Assume a request-latency histogram with 12 buckets plus _sum and _count, which gives 14 series per label set. It is labeled by route, method, status code and pod:

Dimension Distinct values (assumed) Running series total
Histogram series (buckets + _sum + _count) 14 14
route 20 280
method 5 1,400
status_code 8 11,200
pod 50 560,000

In practice the total is lower, because not every route sees every method and status. The worst case is still the right figure for capacity planning, because a deploy, an error spike or a crawler can populate combinations that were empty before.

Now add user_id, or replace the route template with the raw URL path, such as /orders/84213. Neither label has a fixed number of values. Every new user or ID creates another 11,200 series per pod set, so the count grows with traffic instead of leveling off.

Bucket count is a multiplier too. Going from 12 buckets to 6 roughly halves this metric's series. However, histogram_quantile interpolates within buckets, so wider buckets make percentile estimates less accurate.

Newer Prometheus releases offer native (sparse) histograms, which store the bucket distribution within a single series. Confirm that your Prometheus version, remote-write path and storage backend support them before you rely on them.

Which labels cause high cardinality, and when should you avoid them?

Two properties make a label expensive. Total cardinality is how many distinct values exist at once, which sets the active series count and memory use. Churn is how fast old values disappear and new ones appear. Every new value creates a series that keeps using the index until it ages out of the head block, so a label with modest concurrent cardinality can still bloat storage if it rotates constantly.

Label Total cardinality Churn Better home
User or tenant ID Grows with your customer base Low to moderate Logs, traces; a bounded tenant tier label
Request or trace ID Effectively unbounded Every request Traces; exemplars
Raw URL path with IDs Unbounded High Route template label (/orders/:id)
Full error message Unbounded (timestamps, IDs in text) High Logs; a short error-class label
Client IP address Very large High Logs
Pod or container name Bounded per moment High under autoscaling and frequent deploys Keep only if per-instance alerting needs it; otherwise aggregate it away
Build or version string Small at any moment Each release adds values Keep a single info metric, not a label on every series

Use this decision rule: a dimension belongs in a metric label only if its value set is bounded and known ahead of time, and only if you will aggregate, alert or build dashboards on it. If you only need it to investigate a single event, put it in logs or traces. Those systems are built for high-cardinality lookups and do not charge you for an index entry per value.

How do you find which metrics and labels have the most series?

Before changing anything, measure. Prometheus already tracks where its series come from, and the cheapest place to look is the TSDB Status page under the Status menu of the web UI. It reads from the head block's index, so it costs very little. It lists the metric names with the most series, the label names with the most distinct values, and the label-value pairs that appear in the most series. Usually one or two metrics account for a large share of the total, and this page shows them first.

For more detail, use PromQL. count by (__name__)({__name__=~".+"}) returns a series count per metric name. Once you have a suspect metric, count(count by (user_id) (http_requests_total)) tells you how many distinct values one label contributes to it. Both queries touch every matching series. On a large instance they can use enough memory to slow down or crash the server. Run them against a replica or during low traffic, and limit the time range to an instant query.

To rank fixes, multiply each metric's series count by the share of its labels you plan to drop or bucket. Illustrative estimate: a metric with 400,000 series whose path label holds 2,000 raw URLs could drop to about 4,000 series if those URLs collapse into 20 route templates. Cutting a 5,000-series metric to zero saves less than one percent of that. Start with the fix that removes the most series.

#!/usr/bin/env bash
# Inspect series cardinality via the Prometheus HTTP API.
# Point this at a replica, not the primary serving dashboards and alerts.
set -euo pipefail

PROM=""   # placeholder host

# 1. Cheap: TSDB status reads head-block index stats (same data as Status > TSDB Status).
echo "== Top metrics by series count (head block) =="
curl -s "$PROM/api/v1/status/tsdb" \
  | jq -r '.data.seriesCountByMetricName[] | "\(.value)\t\(.name)"'

echo "== Labels with the most distinct values =="
curl -s "$PROM/api/v1/status/tsdb" \
  | jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"'

# 2. Expensive: these queries touch every matching series.
# Instant queries only (/api/v1/query), never range queries, and run at low traffic.
echo "== Series per metric name (top 10) =="
curl -s -G "$PROM/api/v1/query" \
  --data-urlencode 'query=topk(10, count by (__name__)({__name__=~".+"}))' \
  | jq -r '.data.result[] | "\(.value[1])\t\(.metric.__name__)"'

# 3. For a suspect metric, count distinct values of one label.
METRIC="http_requests_total"
LABEL="path"
echo "== Distinct values of '$LABEL' in $METRIC =="
curl -s -G "$PROM/api/v1/query" \
  --data-urlencode "query=count(count by ($LABEL) ($METRIC))" \
  | jq -r '.data.result[0].value[1]'

# Rank fixes: series_count * fraction removed by dropping or bucketing the label.
# A metric near the top of list 1 with a high count in step 3 is the first target.

A shell script that ranks metrics and labels by series count, first using the low-cost TSDB status endpoint and then using the heavier PromQL counting queries as instant queries against a replica.

How do you fix high cardinality at the instrumentation source?

The cheapest series is the one your code never emits. Label the route template your router matched (/users/{id}), not the raw path; most HTTP frameworks expose the template after routing. If dashboards only need to know whether requests are failing, record status_class="5xx" instead of the exact code. That gives five values instead of dozens. Request IDs, user IDs and trace IDs belong in logs or span attributes, which are designed to store data per event. To get from a latency spike to one specific request, attach an exemplar. The client library stores a trace ID with a sampled observation, outside the label set, so it creates no new series. Exemplar storage has to be enabled on the Prometheus server for this to work.

A team rule for unbounded values

Adopt one rule: no label on a shared metric may take values from an unbounded set, such as IDs, raw paths, free-form error strings or emails. Any new label on a shared metric goes through review with an estimate of how many values it can take. The reviewer multiplies that estimate by the existing label combinations.

Trade-offs

Dropping a dimension is permanent for data already ingested without it. You cannot later split last quarter's data by customer. Moving a dimension to logs or traces shifts the cost to those systems rather than removing it, and ad hoc aggregation there is slower than in PromQL. Those systems need their own budget.

The payoff is engineering time. Rejecting a label in review costs one comment. Finding the same label in production costs an incident, new relabel rules, and dashboards and alerts that have to be rewritten.

import logging
from prometheus_client import Histogram

log = logging.getLogger("http")

# Every label below has a small, known set of values.
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency by route template and status class.",
    ["method", "route", "status_class"],
)

def status_class(code: int) -> str:
    # Map 200, 404, 503, etc. to 2xx/4xx/5xx: five values instead of dozens.
    return f"{code // 100}xx"

def record_request(method, route_template, status_code, duration_s,
                   request_id, user_id, trace_id):
    # route_template is the pattern the router matched, e.g. "/users/{id}".
    # Never pass the raw path. Use one fixed value for requests with no match.
    route = route_template or "unmatched"

    REQUEST_LATENCY.labels(
        method=method,
        route=route,
        status_class=status_class(status_code),
    ).observe(
        duration_s,
        # The exemplar is stored alongside a sampled observation, not in the
        # label set, so it creates no new series. It is exposed only in the
        # OpenMetrics format, and the Prometheus server needs
        # --enable-feature=exemplar-storage to keep it.
        exemplar={"trace_id": trace_id},
    )

    # Per-request identifiers go to the log, which stores one record per event.
    log.info("request done", extra={
        "request_id": request_id, "user_id": user_id, "trace_id": trace_id,
        "path_template": route, "status": status_code,
    })

A prometheus_client histogram labeled only by method, route template and status class, with the trace ID attached as an exemplar and per-request IDs sent to logs instead of labels.

How does Prometheus relabeling reduce cardinality at scrape time?

Relabeling happens in two places, and only one of them affects cardinality. relabel_configs runs during service discovery, before the scrape. It sees target labels such as __address__ and __meta_* and decides which targets to scrape and what labels they carry. metric_relabel_configs runs after the scrape and before ingestion, on every sample, so it is where per-metric cardinality fixes go.

The common actions under metric_relabel_configs:

Action Example settings Effect
labeldrop regex: pod_template_hash Removes matching label names from every sample
labelkeep regex: __name__|job|instance|method|code Removes every label not matched. Omit __name__ and the metric is broken.
drop source_labels: [__name__], regex: http_request_duration_seconds_bucket Discards whole series before storage
replace source_labels: [path], regex: /users/[0-9]+, target_label: path, replacement: /users/:id Collapses many values into one. The regex is fully anchored.

These fixes have costs. The target still exposes every series, so network bandwidth and parse CPU are spent on data you throw away. Dropping a label or collapsing values can also merge series that used to be distinct. If two merged samples arrive in the same scrape, Prometheus rejects them as duplicates. If merged counters come from different sources, they interleave, and rate() reads the drops as counter resets.

Use relabeling for third-party exporters you cannot change, and as an emergency brake during a cardinality incident. For code you own, treat a relabel rule as a stopgap: open a ticket to fix the instrumentation, then delete the rule.

scrape_configs:
  - job_name: payments-api
    kubernetes_sd_configs:
      - role: pod
    # Runs BEFORE the scrape, on target labels: decides which pods are scraped.
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
    # Runs AFTER the scrape, on every sample, before it reaches the TSDB.
    # Rules apply in order, so collapse values before labelkeep runs.
    metric_relabel_configs:
      # Remove a label whose values change on every deployment.
      - action: labeldrop
        regex: pod_template_hash
      # Discard an entire high-cardinality histogram.
      - source_labels: [__name__]
        action: drop
        regex: http_request_duration_seconds_bucket
      # Collapse /users/123 and /users/123/orders into one templated value.
      # The regex is anchored, so (/.*)? is needed to match subpaths.
      - source_labels: [path]
        regex: /users/[0-9]+(/.*)?
        target_label: path
        replacement: /users/:id$1
        action: replace
      # Allowlist label names. Keep __name__, and keep le so the
      # histograms that survive still work.
      - action: labelkeep
        regex: __name__|job|instance|method|code|path|le

A scrape job that filters targets with relabel_configs, then uses metric_relabel_configs to drop a label, discard a histogram, collapse path values with an anchored regex and allowlist the remaining labels.

What are the alternatives: recording rules, aggregation-only pipelines and guardrails?

Recording rules precompute aggregates such as per-service request rates, so dashboards and alerts query a few series instead of fanning out over thousands. They do not reduce cost by themselves. Each rule writes new series and adds evaluation work on every interval, so a rule layered over untouched raw data makes the bill slightly worse. The savings come from what you do next. You can drop the raw series after aggregation or keep them on short local retention. You can also federate or remote-write only the aggregated series, for example by filtering on a rule-name prefix, so that long-term storage never holds the expensive raw data.

Guardrails limit the damage when a fix arrives too late. In scrape config, sample_limit caps samples per scrape, and label_limit with its length variants rejects pathological label sets. When a target exceeds sample_limit, Prometheus fails the entire scrape, not just the excess samples. That target goes dark exactly when it is misbehaving. Set limits well above normal output, and alert on prometheus_target_scrapes_exceeded_sample_limit_total increasing. Separately, alert on sustained growth of prometheus_tsdb_head_series so you catch slow drift before memory runs out.

Approach Where cost is removed What is lost Engineering effort
Source fix Client, network, ingest, storage Nothing that was needed High: code changes, deploys
Relabeling Ingest and storage Dropped labels or series, silently Low per rule, but ongoing upkeep
Recording rules + raw drop/short retention Long-term storage, query time Raw detail beyond retention Medium: rule design, query migration
Limits and alerts Prevents runaway cost Whole scrapes when a limit fires Low, but needs tuning

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to prometheus high cardinality, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.