What is Prometheus high cardinality and why does it multiply?

In Prometheus, a time series is identified by its metric name plus its complete set of label key-value pairs. http_requests_total{method="GET", route="/orders/:id", status="2xx"} and the same metric with status="5xx" are two separate series. Each one gets its own entry in the in-memory head block, which holds recent samples along with index and label data for every active series. Memory use therefore grows with the number of distinct series.

This is why cardinality multiplies. The upper bound on series for a metric is the product of the distinct values of each label.

Illustrative estimate (assumed inputs, not measured):

  • method: 5 values
  • route (templated, e.g. /orders/:id): 40 values
  • status (class, e.g. 2xx): 5 values

5 × 40 × 5 = 1,000 possible series per scrape target. Across 20 instances, that is up to 20,000. Not every combination will occur, but this is the ceiling you plan capacity against.

Now add user_id with an assumed 10,000 active users. The ceiling becomes 1,000 × 10,000 = 10 million series per instance. One unbounded label turns a manageable metric into one that can exhaust the server's memory.

Churn is a second source of growth. Labels such as pod or instance take new values every time a pod is rescheduled or redeployed. The old series stop receiving samples, but they stay in the head block and index until they age out. A cluster that redeploys often can carry several generations of series simultaneously, even if its steady-state cardinality looks modest.

 METRIC: http_requests_total          (illustrative estimate, assumed inputs)
 +-----------+   +-------------+   +-------------+
 | method    |   | route       |   | status      |
 | 5 values  | x | 40 values   | x | 5 values    |  = 1,000 series/target
 +-----------+   +-------------+   +-------------+  x 20 instances = 20,000
                        |
                        v  add user_id (10,000 values)
          +----------------------------------+
          | 1,000 x 10,000 = 10,000,000      |
          | series ceiling per instance      |
          +----------------------------------+
                        |
                        v  each unique label set = 1 series
 +-----------------------------------------------------------------+
 | PROMETHEUS HEAD BLOCK (in memory)                               |
 |  {method="GET",route="/orders/:id",status="2xx"}  -> samples    |
 |  {method="GET",route="/orders/:id",status="5xx"}  -> samples    |
 |  ...  + index and label data for every active series            |
 +-----------------------------------------------------------------+
                        ^
                        |  churn: new pod label value per redeploy
 +--------------+   +--------------+   +--------------+
 | pod=api-a1   |-->| pod=api-b7   |-->| pod=api-c3   |
 | stale, held  |   | stale, held  |   | active       |
 | until age-out|   | until age-out|   |              |
 +--------------+   +--------------+   +--------------+

Label value counts multiply into the series ceiling, so a single unbounded label like user_id inflates it 10,000-fold, while pod churn leaves stale series in the head block until they age out (illustrative estimate, assumed inputs).

Which labels cause high label cardinality?

The question to ask of any label is whether you could write down every value it will ever take. If you can, it is bounded. If the set grows with traffic, users or time, it is unbounded, and every new value creates a new series for every metric that carries the label.

Unbounded (avoid as labels) Bounded alternative
User ID, email, account ID None; put it in logs or traces
Request ID, trace ID Exemplars or traces
Raw URL path (/orders/8812) Route template (/orders/{id})
Client or pod IP address Zone, node pool, or nothing
Error message string Error class or error code
Status code from arbitrary upstreams Status class (2xx, 4xx, 5xx)
Tenant ID, container ID in a growing fleet Tenant tier, workload name

Raw paths and error strings are the most common surprises. They look harmless in a test environment with ten requests, but in production a crawler hitting random URLs or a parser embedding input in its error message can generate thousands of values in an hour.

Some labels seem bounded and still churn. Pod names, container IDs and instance addresses are finite at any moment, but each rollout replaces them. A deployment with 50 pods that ships twice a day produces 100 new values daily, each carrying its own copy of every series. This churn is the multi-generation effect described above.

Anyone adding instrumentation needs to apply this test, because the person who adds the label rarely pays the cost. On a shared Prometheus server, one team's careless label slows queries and consumes memory for everyone else, so platform owners should review label choices too.

How do you find high-cardinality metrics in Prometheus?

Prometheus records enough about its own data to find offenders without any external tooling. Start with a ranking by metric name: topk(10, count by (__name__)({__name__=~".+"})) returns the metrics that own the most series. This query touches every series in the head block, so run it sparingly on a large server, and prefer an instant query over a range query.

Once you have a suspect metric, find out which label is driving it. count(count by (path)(http_requests_total)) gives the number of distinct path values. Repeat the query for each label on the metric. The label whose distinct count tracks requests, users or pods rather than a small fixed set is usually the problem.

The TSDB status page in the web UI (Status → TSDB Status) shows the same picture without writing queries. It lists the metric names with the most series, the label names with the most distinct values, and the label pairs that use the most memory. It reads from the head block only, so it reflects recent data, not total history.

Cardinality also changes over time, and a single snapshot can't show that. Graph prometheus_tsdb_head_series alongside process_resident_memory_bytes to catch steady growth. Also graph rate(prometheus_tsdb_head_series_created_total[5m]) to catch churn: series that appear and disappear quickly, for example on every pod restart. Churn inflates memory and the index even when the live series count looks flat.

When to run these checks:

  • Routinely, before capacity problems appear. Growth is cheap to fix while it is gradual.
  • After every deployment, when new labels first show up.
  • During incident review, whenever Prometheus itself was slow or ran out of memory.
#!/usr/bin/env bash
# Cardinality check against a Prometheus server using only its HTTP API.
# Run routinely, after deployments, and during incident review.
set -euo pipefail

PROM="${PROM:-"

# Helper: run an instant query (cheaper than a range query) and print metric + value.
q() {
  curl -sG "$PROM/api/v1/query" --data-urlencode "query=$1" \
    | jq -r '.data.result[] | "\(.metric | tostring)\t\(.value[1])"'
}

echo "== Top 10 metrics by series count (touches every head series; run sparingly) =="
q 'topk(10, count by (__name__)({__name__=~".+"}))'

echo "== Distinct values of the 'path' label on a suspect metric =="
q 'count(count by (path)(http_requests_total))'

echo "== Live head series, memory, and churn (series created per second) =="
q 'prometheus_tsdb_head_series'
q 'process_resident_memory_bytes{job="prometheus"}'
q 'rate(prometheus_tsdb_head_series_created_total[5m])'

# Same data as Status -> TSDB Status in the web UI (head block only).
echo "== TSDB status: top metric names and label names =="
curl -s "$PROM/api/v1/status/tsdb" | jq '{
  headSeries: .data.headStats.numSeries,
  topMetrics: .data.seriesCountByMetricName,
  topLabelsByValueCount: .data.labelValueCountByLabelName,
  topLabelsByMemory: .data.memoryInBytesByLabelName
}'

A bash script that queries Prometheus's own HTTP API to rank metrics by series count, count distinct values of a suspect label, check head series, memory and churn, and pull the same data the TSDB Status page shows.

How should you fix cardinality at instrumentation time?

The most effective fix happens before a sample is ever scraped. Relabeling can drop a bad label, but it cannot repair a metric whose design is wrong. That makes instrumentation code the place to fix cardinality.

  • Label routes, not paths. Record the router's template, such as /users/{id}, instead of /users/48213. The template set is finite and known at build time. Raw paths grow with traffic.
  • Bucket status codes. 2xx, 4xx and 5xx usually answer the alerting question. Keep exact codes only where a specific one, such as 429, drives a decision.
  • Move per-entity detail out of metrics. User IDs, order IDs and tenant keys belong in logs or trace attributes. Those systems are built for that kind of lookup.
  • Treat each new label as a design decision. Write down its maximum value count in the review, and reject labels whose bound is "unknown."

Histograms need extra care. Every bucket is its own series, plus _sum and _count, so each label multiplies by the bucket count.

Illustrative estimate (assumed numbers, not a benchmark): series per target = routes × methods × status values × (buckets + 2)

  • 40 × 4 × 5 × (12 + 2) = 11,200
  • With 20 exact status codes instead of 5 classes: 44,800
  • With buckets trimmed to 8: 8,000

Keep histogram label sets narrower than counter label sets, and drop buckets that no SLO or dashboard reads.

This approach has a development cost. Service owners have to agree on label conventions, and every fix needs a redeploy. In return, it removes a recurring class of incidents, which saves more engineering time than repeated firefighting.

import logging
import time

from flask import Flask, g, request
from prometheus_client import Histogram, start_http_server

app = Flask(__name__)
log = logging.getLogger("requests")

# Narrow, bounded label set, written down at review time:
# route ~40 templates x method ~4 x status_class 5 values.
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency by route template",
    ["route", "method", "status_class"],
    # Only the buckets the 300ms SLO and dashboards read; +Inf is added automatically.
    buckets=(0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5),
)

@app.before_request
def start_timer():
    g.start = time.perf_counter()

@app.after_request
def record(response):
    # url_rule.rule is the template ("/users/<int:user_id>"), not the raw path.
    # Unmatched requests share one fixed value instead of creating new series.
    route = request.url_rule.rule if request.url_rule else "unmatched"
    status_class = f"{response.status_code // 100}xx"  # 2xx, 4xx, 5xx
    REQUEST_LATENCY.labels(route, request.method, status_class).observe(
        time.perf_counter() - g.start
    )
    # Per-entity detail (raw path, exact code) goes to logs, not labels.
    log.info("path=%s status=%d", request.path, response.status_code)
    return response

@app.route("/users/<int:user_id>")
def get_user(user_id):
    return {"id": user_id}

if __name__ == "__main__":
    start_http_server(8000)  # expose /metrics on a separate port
    app.run(port=8080)

A Flask service records request latency with the route template and status class as labels and a trimmed bucket list, and sends raw paths and exact status codes to logs instead of turning them into labels.

How does relabeling reduce cardinality at scrape time?

Relabeling is the fallback when a fix in code is slow or the code belongs to someone else, such as a third-party exporter. Prometheus has two relabeling stages, and only one of them sees metric labels.

Stage Config block Operates on Runs Typical cardinality use
Target relabeling relabel_configs Discovered targets and their metadata labels Before the scrape Skip targets entirely, control target-level labels
Metric relabeling metric_relabel_configs Every scraped sample After the scrape, before storage Drop series, drop labels, bound label values

Most cardinality work happens in metric_relabel_configs. It supports three main patterns:

  • Drop whole series. Use the drop action with a regex on __name__, for example to remove a histogram's per-bucket series you never query.
  • Drop or replace a label. Use labeldrop to remove a label such as request_id. Use replace to overwrite it with a constant.
  • Bound the values. Use a replace rule with a regex to collapse values. For example, rewrite /users/8812/orders into /users/:id/orders, or map any status code to 2xx, 4xx or 5xx.

The trade-offs are real:

  • Dropped detail is gone permanently. You cannot recover it later.
  • Scrape cost stays the same. The exporter still generates the full payload, and Prometheus still transfers and parses it. Only storage and memory benefit.
  • Dropping a label does not aggregate. If two series differ only by the dropped label, they collapse to identical label sets in the same scrape. Prometheus does not sum them. It ingests one sample, rejects the others as duplicates and flags the scrape with errors.
  • Central rules drift from the code. When a developer renames a label or adds a route pattern, the regex silently stops matching. Keep these rules in version control next to the service and test them.
scrape_configs:
  - job_name: checkout-api
    static_configs:
      - targets: ['checkout-api.internal:9100']

    # Runs after the scrape, before storage. The exporter still emits
    # and Prometheus still parses the full payload; only TSDB load shrinks.
    metric_relabel_configs:
      # 1. Drop whole series: per-bucket histogram series nobody queries.
      #    Regexes are fully anchored, so this matches the exact name only.
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop

      # 2. Remove an unbounded label. labeldrop matches label NAMES.
      #    If two series differ only by request_id, they collide in the
      #    same scrape and the extra samples are rejected as duplicates.
      #    They are not summed.
      - regex: 'request_id'
        action: labeldrop

      # 3a. Bound values: collapse numeric IDs in the path label.
      #     If the regex does not match, the label is left unchanged.
      - source_labels: [path]
        regex: '/users/[0-9]+/orders'
        target_label: path
        replacement: '/users/:id/orders'
        action: replace

      # 3b. Bound values: map status codes to classes such as 2xx and 5xx.
      - source_labels: [code]
        regex: '([1-5])[0-9][0-9]'
        target_label: code
        replacement: '${1}xx'
        action: replace

      # Replace a label with a constant. With no source_labels, the
      # default regex (.*) matches the empty string, so this always applies.
      - target_label: pod_ip
        replacement: 'redacted'
        action: replace

A metric_relabel_configs block that drops an unused histogram series, removes an unbounded label, collapses path and status-code values into bounded sets, and overwrites a label with a constant, with comments marking where series collisions and unchanged scrape cost still apply.

When should you use sample limits, label limits and series-growth alerts?

Relabeling handles the problems you know about. Limits protect you from the ones you don't, such as a new deploy that starts emitting a user ID as a label. They are guardrails, not a fix.

Per-scrape limits

In a scrape config, sample_limit caps how many samples a target may return after metric relabeling. label_limit, label_name_length_limit and label_value_length_limit reject series whose label sets are too large or too long. When a target breaks one of these limits, Prometheus fails that scrape and ingests nothing from it. The target's up series goes to 0, and the rest of the server keeps working. The cost of a cardinality explosion stays with the job that caused it, so it does not exhaust the server's memory and take every other target with it.

The downside is that the failure is all-or-nothing. A target that goes one sample over its limit disappears entirely, often at the moment it is misbehaving and you most need its metrics. Set limits from observed baselines. Look at scrape_samples_post_metric_relabeling per job over a period that includes peak traffic and recent deploys, then add enough headroom for normal growth. Doubling the baseline is a reasonable starting point. Alert on up == 0 and on prometheus_target_scrapes_exceeded_sample_limit_total so that a tripped limit pages someone instead of failing silently.

Series-growth alerts

Limits work per target. To catch slow growth across many targets, alert on prometheus_tsdb_head_series. Use one alert for an absolute ceiling based on the server's memory, and another for an unusual rate of increase over a few hours. The rate alert catches a gradual leak early, while you can still fix it with relabeling before you have to raise limits.

What are the alternatives: recording rules, logs and traces?

Not every cardinality problem has to be solved by deleting labels. Two other options move the cost somewhere else.

Recording rules precompute aggregates on a schedule and store the result as new, smaller series. For example, a rule that sums request rate by service and status code drops the per-pod and per-route dimensions. Dashboards then read a few dozen series instead of fanning out across thousands, so queries get faster and cheaper. The raw series are still scraped, indexed and held in memory, so ingestion cost does not change. Rules also add evaluation work every interval and create their own output series. The biggest risk is that they hide the problem. Dashboards stay fast while the raw metric keeps growing underneath, so the limits and growth alerts from the previous section still need to cover the source metrics.

Logs and traces are the right home for per-entity detail such as user IDs, request IDs or full URLs. They handle unbounded values without a series index. The cost moves to log storage and trace backends, and the queries change too: you search and filter events instead of running cheap numeric aggregations. Trace sampling also means you lose completeness.

Approach Where it acts What it costs What it saves
Instrumentation fix Application code Code changes, deploys Ingestion, storage, query
Relabeling Scrape time Config upkeep, lost detail Ingestion, storage
Sample/label limits Scrape time Failed scrapes when tripped Protects server memory
Recording rules After ingestion Evaluation load, extra series Query cost only
Logs/traces Other systems Their storage, harder queries Prometheus series entirely

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to prometheus high cardinality, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.