Prometheus High Cardinality: Finding and Dropping Expensive Labels

How Prometheus high cardinality works and why it costs TSDB memory

In Prometheus, a time series is identified by its metric name plus its full set of label key-value pairs. Cardinality is the number of unique combinations that exist. Change any single label value and you have created a new series, with its own storage and index entries.

Labels multiply rather than add. Each label's distinct values are a factor in the total.

Illustrative estimate (assumed numbers, not a benchmark):

• Formula: series ≈ buckets × status codes × paths
• Applied: 10 × 5 × 2,000 ≈ 100,000 series for one histogram

The _sum and _count series add a little more on top. Swap the 2,000 raw paths for 20 route templates and the same metric drops to about 1,000 series.

Why this costs memory

Recently written series live in the TSDB head block, which is held in memory. Each active series carries:

• its label set
• entries in the inverted index that maps label pairs to series
• its most recent chunks of samples

Memory therefore scales with the number of series, not the number of samples.

Churn makes this worse. When short-lived series appear and disappear, each one still occupies the head and the index until the head is compacted. Examples include pods replaced on every deploy, job IDs and per-request values. A snapshot of active series can look moderate while the head is carrying many times that number.

In practice, the problem is rarely the total series count. It is usually a few unbounded labels: user IDs, raw URL paths, pod names under heavy churn, or free-text error messages. Finding and bounding those few labels is most of the work.

Copy http_request_duration_seconds_bucket{le, status, path}
 +-----------+   +--------------+   +---------------+
 | le        |   | status       |   | path (raw)    |
 | 10 values | x | 5 values     | x | 2,000 values  |
 +-----------+   +--------------+   +---------------+
       |                |                   |
       +----------------+-------------------+
                        |
                        v
        +-------------------------------+
        | ~100,000 unique series        |
        | (illustrative estimate)       |
        +-------------------------------+
                        |
                        v
 +------------------------------------------------------+
 | TSDB head block (in memory)                          |
 |  per active series:                                  |
 |  +------------+  +------------------+  +-----------+ |
 |  | label set  |  | inverted index   |  | recent    | |
 |  |            |  | label -> series  |  | chunks    | |
 |  +------------+  +------------------+  +-----------+ |
 +------------------------------------------------------+
        ^                               |
        |                               v
 +----------------+            +------------------+
 | churn: new pod |            | held until head  |
 | / job / req ID |            | compaction       |
 +----------------+            +------------------+

The distinct values of each label multiply into roughly 100,000 series for one histogram (illustrative estimate). Every active series keeps its label set, index entries and recent chunks in the in-memory head block, and series created by churn stay there until the head is compacted.

Who should care and when to act

Two groups end up owning this problem. Backend engineers who instrument services decide which labels exist. A single labels(user_id) call in a request handler can add millions of series. Platform engineers who run Prometheus absorb the consequences: memory sizing, retention trade-offs, and the pager when the server falls over. Neither group can fix it alone. The platform team can see which metrics are expensive, but only the service owner knows whether a label carries information anyone queries.

Signals worth investigating

• Head memory climbing steadily with no matching growth in scrape targets. Every active series in the head block holds memory, so growth without new targets usually means new label values.
• Queries and dashboards slowing down, especially aggregations over a single metric name. A query has to touch every matching series, so more series means more work.
• OOM kills and restart loops. Replaying the write-ahead log on startup rebuilds the same oversized head, so the server can crash again during recovery.
• A step change in series count right after a deploy. This is the clearest signal. It points to a specific commit that added or changed a label.

Not all cardinality is waste

A pod label in a 2,000-pod cluster produces many series, and it earns them. It lets you isolate one misbehaving replica. Stripping it to save memory would make incidents harder to debug. The target is bounded, intentional cardinality: every label should have a known, finite set of values that someone actually uses. A low series count is not the goal.

Finding the expensive metrics and labels

Start with the TSDB status page (Status → TSDB Status in the web UI) or its JSON form at /api/v1/status/tsdb. Both read statistics Prometheus already keeps for the in-memory head block, so they cost almost nothing to query. They are the right first step even on a server that is close to running out of memory. The page lists three things:

• Top metric names by series count. This shows which metric to look at.
• Top label names by distinct value count. This shows which label is unbounded.
• Label-value pairs with the most series. This often points to a single job or instance.

Once you have a suspect, PromQL can confirm it. topk(10, count by (__name__)({__name__=~".+"})) ranks metrics by series count. Be careful with it, because that selector matches every series in the head and can push a struggling server over the edge. Run it sparingly, or scope it with a matcher such as job="api". For a single metric, count(count by (label_name)(metric_name)) returns how many distinct values a label has. A label that reports thousands of values is a candidate for removal.

To keep the problem from coming back, alert on growth:

• prometheus_tsdb_head_series tracks total active series.
• rate(prometheus_tsdb_head_series_created_total[5m]) shows series churn. Churn can be high even when the total looks flat, for example when pod names or request IDs constantly create and abandon series.
• scrape_series_added is recorded per target. A spike in it right after a deploy points to the service that shipped a new label.

Copy#!/usr/bin/env bash
# Drill down from the cheap TSDB status endpoint to targeted PromQL checks.
set -euo pipefail
PROM=""

# Step 1: read precomputed head-block stats. This is cheap even on a struggling server.
curl -s "$PROM/api/v1/status/tsdb" | jq '{
  head: .data.headStats,
  top_metrics: .data.seriesCountByMetricName,
  top_labels_by_values: .data.labelValueCountByLabelName,
  top_label_pairs: .data.seriesCountByLabelValuePair
}'

# Helper: run an instant query and print each result's labels and value.
q() {
  curl -s -G "$PROM/api/v1/query" --data-urlencode "query=$1" \
    | jq -r '.data.result[] | "\(.metric | tostring) \(.value[1])"'
}

# Step 2: rank metrics by series count, scoped to one job instead of {__name__=~".+"}.
q 'topk(10, count by (__name__)({job="api"}))'

# Step 3: count distinct values of one suspect label on one suspect metric.
q 'count(count by (path)(http_requests_total{job="api"}))'

# Step 4: ongoing signals for total series, series churn and per-target growth.
q 'prometheus_tsdb_head_series'
q 'rate(prometheus_tsdb_head_series_created_total[5m])'
q 'topk(5, scrape_series_added)'

A shell script that reads the TSDB status endpoint first, then runs scoped PromQL queries to find the expensive metric and label and to check the series-growth signals.

Fixing cardinality at the source

The cheapest series is the one never created. Once you know which label is exploding, change the instrumentation that emits it rather than filtering it downstream. The rule is simple: every label value should come from a small, known set.

In practice that means replacing unbounded values with bounded ones:

• Raw paths and URLs become route templates. /users/8812/orders/55 becomes /users/{id}/orders/{id}.
• Exact status codes can become status classes (2xx, 4xx, 5xx) when dashboards and alerts only ask about classes.
• Customer or tenant IDs become tenant tiers or plans, which is usually what capacity and SLO questions actually need.
• Error messages, SQL, stack traces and user input never belong in labels. They are free text, so each new variation creates a new series. Use a fixed error_type instead.

Here is a typical before-and-after. A handler records http_requests_total{path="/users/8812/orders/55"}, which creates one series per user and order. The fixed version reads the template from the router and records http_requests_total{route="/users/{id}/orders/{id}"}. Series count now grows with the number of routes, not with traffic.

You don't lose the per-request detail. Put it in logs or trace attributes, where high-cardinality data is expected and stored per event rather than kept active in TSDB memory.

Fixing the source also saves engineering time later. Relabeling protects only the Prometheus server that has the rule. A source fix protects every consumer: the exporter stops generating and serving expensive series, scrapes get smaller, and the next team that points a scraper at the endpoint won't have to rediscover and patch the same problem.

Copyfrom flask import Flask, request
from prometheus_client import Counter, start_http_server

app = Flask(__name__)

# BEFORE (unbounded): one series per user, order, query string and typo.
# REQUESTS = Counter("http_requests_total", "HTTP requests", ["method", "path"])
# REQUESTS.labels(request.method, request.path).inc()
#   -> path="/users/8812/orders/55", path="/users/8813/orders/56", ...

# AFTER (bounded): label values come from small, known sets.
REQUESTS = Counter(
    "http_requests_total", "HTTP requests", ["method", "route", "status_class"]
)

@app.after_request
def record(response):
    # url_rule holds the matched route template, not the raw path.
    # Unmatched requests (404s, scanners) collapse into one value.
    route = request.url_rule.rule if request.url_rule else "unmatched"
    # 200 -> "2xx", 404 -> "4xx": enough for dashboards and SLO alerts.
    status_class = f"{response.status_code // 100}xx"
    REQUESTS.labels(request.method, route, status_class).inc()
    return response

@app.route("/users/<int:user_id>/orders/<int:order_id>")
def get_order(user_id, order_id):
    # Per-request detail (IDs, error text) goes to logs or traces, not labels.
    app.logger.info("order lookup user_id=%s order_id=%s", user_id, order_id)
    return {"user_id": user_id, "order_id": order_id}

if __name__ == "__main__":
    start_http_server(9100)  # serves /metrics on a separate port
    app.run(port=8080)
    # Emits: http_requests_total{method="GET",
    #   route="/users/<int:user_id>/orders/<int:order_id>",status_class="2xx"}

A Flask handler instrumented with prometheus_client that replaces the raw request path label with the matched route template and a status class, and sends per-request IDs to logs instead.

Dropping and bounding labels with metric_relabel_configs

When an exporter can't be fixed quickly, the scrape config is the backstop. These rules go in metric_relabel_configs, not relabel_configs. relabel_configs runs against targets before the scrape and only sees target labels such as __address__. Metric names and the labels an exporter emits don't exist yet at that stage. metric_relabel_configs runs on every scraped sample before ingestion, so it can act on them.

Three actions cover most cases:

• drop on __name__ removes whole metrics, for example an unused histogram whose buckets multiply every label.
• labeldrop removes a label by name across all series in the job.
• replace with a capturing regex collapses values, such as rewriting /users/12345 to /users/:id or keeping only the status class 5xx.

Use labeldrop carefully. If removing a label leaves two series with identical label sets, Prometheus rejects the duplicates within that scrape and you lose samples silently. Drop only labels that are truly redundant, such as a copy of the instance name. If the label distinguishes real series, collapse it or aggregate before dropping.

To verify, compare scrape_samples_scraped with scrape_samples_post_metric_relabeling for the target, and query the affected metric in the UI to confirm its label set.

For guardrails, set sample_limit and the label limits (label_limit, label_name_length_limit, label_value_length_limit) per job. Exceeding sample_limit fails the entire scrape and sets up to 0, so alert on that condition and on prometheus_target_scrapes_exceeded_sample_limit_total.

Finally, watch prometheus_tsdb_head_series drop and level off. Series that are already in the head stay there until head compaction removes them, and old blocks remain on disk until retention deletes them.

Copyscrape_configs:
  - job_name: myapp
    static_configs:
      - targets: ['myapp-1.internal:8080', 'myapp-2.internal:8080']

    # Guardrails: if the target exceeds sample_limit after relabeling,
    # the whole scrape fails and up becomes 0.
    sample_limit: 50000
    label_limit: 30
    label_name_length_limit: 128
    label_value_length_limit: 512

    # Runs on scraped samples, after the scrape and before ingestion.
    metric_relabel_configs:
      # Remove an unused histogram whose buckets multiply every label.
      - source_labels: [__name__]
        regex: 'myapp_request_duration_seconds_bucket'
        action: drop

      # Collapse numeric IDs: /users/12345 -> /users/:id
      # The regex is anchored, so it must match the whole value.
      - source_labels: [path]
        regex: '/users/[0-9]+'
        target_label: path
        replacement: '/users/:id'
        action: replace

      # Keep only the status class: 503 -> 5xx
      # ${1} is braced so that 'xx' is not read as part of the group name.
      - source_labels: [code]
        regex: '([1-5])[0-9]{2}'
        target_label: code
        replacement: '${1}xx'
        action: replace

      # Drop only a redundant label, here a copy of the instance name.
      # Dropping a distinguishing label creates duplicate series.
      - regex: 'exporter_hostname'
        action: labeldrop

# Verify: scrape_samples_scraped vs scrape_samples_post_metric_relabeling
# Alert on: up == 0, increase(prometheus_target_scrapes_exceeded_sample_limit_total[5m]) > 0

A scrape job that uses metric_relabel_configs to drop a histogram, collapse path and status-code values, and remove a redundant label, with sample and label limits as guardrails.

When to avoid relabeling and what the alternatives are

Each of these levers has a cost, and some of them fail in ways that are hard to notice.

• Scrape-time drops are permanent. Prometheus never stores the dropped data, so you cannot backfill it later. Before you remove anything, check with the teams whose dashboards, alerts and SLO queries use that metric.
• Relabeling hides the problem. The exporter still generates every series, serializes it and sends it over the network on each scrape. Prometheus also has to parse each series before it can discard it. Ingestion memory goes down, but CPU and network costs stay the same.
• labeldrop can merge series. If two series become identical once a label is removed, Prometheus accepts one sample per timestamp and rejects the others. The only evidence is a duplicate-sample counter, while your graphs show values that are wrong and look normal.
• Recording rules make queries cheaper, not ingestion. Unless you also drop the raw series, the head holds both the raw series and the new aggregate.
• sample_limit fails the whole scrape. If a target exceeds the limit, all of its metrics are lost for that scrape, including up-adjacent health signals you rely on.
• Churn is a separate problem from label design. Rolling deploys and autoscaling create new pod or instance values, and each value starts a new set of series. Old series stay in the head until compaction removes them, so frequent restarts can dominate head memory even when labels are well designed.

Impact on cost and engineering time

Every active series costs head-block memory, index entries, and work at query time. Queries that aggregate across a high-cardinality metric touch more series, so dashboards get slower and rule evaluation takes longer. When cardinality grows without anyone noticing, the result is an out-of-memory restart and a gap in monitoring, usually during an incident.

Illustrative estimate (assumed numbers, not a benchmark)

Upper-bound series = series per label set × product of distinct values per label.

Assume the following inputs:

• A histogram with 12 buckets plus _sum and _count, giving 14 series per label set.
• 5 methods, 20 routes, 4 status classes, and 30 pods.
• An assumed 4 KB of memory per active series. Measure your own value from the TSDB status page and your process memory.

Real data fills only part of that product. Even so, the multiplier is set by the unbounded label, not by traffic.

A repeatable workflow

Teams avoid repeated investigations by following the same steps every time:

• Check the TSDB status page for the top metrics and labels by series count.
• Alert on sudden jumps in scrape_series_added.
• Alert on scrapes rejected for exceeding sample_limit.
• During code review, ask whether each new label value is bounded.

Up-front versus recurring cost

Fixing labels at the source and writing recording rules takes developer time once. The alternatives cost time again and again. A relabel rule keyed to a label name breaks silently when that name changes, and every OOM incident pulls someone off planned work for debugging and a postmortem.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to prometheus high cardinality, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.