What is high cardinality in Prometheus and why does it drive metrics costs?
In Prometheus, cardinality is the number of unique combinations of a metric name and its label values. Each combination is stored as its own time series, with its own index entries, in-memory chunk, and stream of samples. Labels multiply, so series count grows much faster than most people expect.
Illustrative estimate (assumed numbers, not a benchmark)
Formula: series = endpoints × status codes × methods × pods × (any additional label's distinct values)
Labels on http_requests_total |
Assumed distinct values | Worst-case series |
|---|---|---|
| endpoint, status, method, pod | 20 × 5 × 3 × 50 | 15,000 |
| + user_id | × 10,000 active users | 150,000,000 |
The second row is an upper bound. Not every user hits every endpoint on every pod. Even a small fraction of that total, though, is orders of magnitude above the original 15,000. Because user_id is unbounded, the count keeps climbing as new users arrive.
Why this costs you
- Memory and index: every active series lives in the TSDB head block, and every label value becomes an index entry.
- Queries and rules: selectors and aggregations must touch more series, which slows dashboards and recording rules.
- Restarts: WAL replay rebuilds the head, so startup takes longer as series count grows.
- Bills: backends that charge per active series or per ingested sample turn label growth directly into higher metrics costs.
This guide is for backend engineers who write instrumentation and platform engineers who run Prometheus.
How do you find which labels are causing series growth?
Start with the TSDB status page in the Prometheus web UI (under Status). It reports statistics for the head block, meaning recently ingested data. It lists the metric names with the most series, the label names with the most distinct values, and the label-value pairs attached to the most series. Prometheus already tracks these numbers internally, so reading the page costs almost nothing. This matters when the server is already short on memory because of cardinality.
To dig further, use PromQL. count by (__name__) ({__name__=~".+"}) returns the series count for every metric name. To check a suspect label, count(count by (user_id) (http_requests_total)) returns how many distinct user_id values that metric carries. Be careful with the first query. It touches every series in the head and can push an overloaded server into an out-of-memory crash. Prefer the status page, scope queries to a single job or metric where you can, and run broad ones off-peak.
When you read the results, look for concentration. Cardinality is usually lopsided. A handful of metrics often account for a large share of all series, and within those metrics one label typically does most of the multiplying. Common culprits are request paths with embedded IDs, pod or container names that change with every deploy, and raw error strings. Rank metrics by series count, then check which label on each has the most distinct values. Fixing the top few gives most of the reduction for a fraction of the effort a full audit of every metric would take.
#!/usr/bin/env bash
# Find which metrics and labels drive series growth.
# Requires curl and jq. PROM is a placeholder for your Prometheus server.
set -euo pipefail
PROM=""
# 1) Cheap: the same head-block stats the Status > TSDB page shows.
# Prometheus already tracks these, so this is safe on a struggling server.
curl -s "$PROM/api/v1/status/tsdb" > tsdb.json
echo "== Metrics with the most series =="
jq -r '.data.seriesCountByMetricName[] | "\(.value)\t\(.name)"' tsdb.json
echo "== Label names with the most distinct values =="
jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"' tsdb.json
echo "== Label=value pairs attached to the most series =="
jq -r '.data.seriesCountByLabelValuePair[] | "\(.value)\t\(.name)"' tsdb.json
# 2) Expensive: PromQL over many series. Scope by job and run off-peak.
# Top 10 metric names by series count for one job only.
curl -s "$PROM/api/v1/query" \
--data-urlencode 'query=topk(10, count by (__name__) ({job="api-server"}))' \
| jq -r '.data.result[] | "\(.value[1])\t\(.metric.__name__)"'
# 3) For a suspect metric, count distinct values of one label.
# A large number here usually means IDs, paths or error strings in a label.
curl -s "$PROM/api/v1/query" \
--data-urlencode 'query=count(count by (user_id) (http_requests_total))' \
| jq -r '.data.result[0].value[1]'
A script that reads the TSDB status API for low-cost cardinality rankings, then uses scoped PromQL queries to confirm which metric and label are doing most of the multiplying.
Which labels should you keep, bucket, or remove?
Once you know which labels are expensive, judge each one by a single question: what query does this label answer? If you can name a dashboard panel, alert, or recurring investigation that groups or filters by it, the label earns its series. If you can't, it is cost without a reader.
| Category | Examples | Action |
|---|---|---|
| Bounded and useful | method, status class (2xx, 5xx), route template (/users/{id}) |
Keep |
| Bounded but too granular | Raw status code where class would do; pod on metrics only viewed in aggregate |
Bucket or aggregate |
| Unbounded | User IDs, request IDs, raw URLs with path parameters, email addresses, trace IDs, error message strings | Remove |
Unbounded identifiers belong in logs or traces, not metrics. Those systems are built to store per-event detail. Prometheus creates a new series for every distinct value and keeps it in the index, so an unbounded label grows without limit.
Per-pod and per-instance labels need a separate decision. They are often the largest multiplier, because every series is copied once per replica. They are also how you find the one instance that is leaking memory or returning errors while the fleet average looks fine. Decide per metric. Keep instance labels on saturation and error metrics, where one bad replica matters. Consider dropping them from high-volume histograms that you only ever read as a service-wide percentile.
Some labels should not be trimmed even when they cost a lot. If a bounded label is used in an alerting rule, removing or merging it can silently break that alert or make it fire on the wrong grouping. Check rule files and dashboards before you change any label.
How do you fix high-cardinality labels in instrumentation code?
Relabeling hides a problem, but fixing it in the instrumentation removes it. When the code that creates a metric emits bounded values, every later stage stays simpler.
Start with request labels. Label HTTP metrics by the route template the router matched, such as /users/{id}, not the raw path /users/48213. Most web frameworks expose the matched pattern, so use it. Do the same with status codes and record 2xx, 4xx, 5xx instead of every individual code, unless someone actually alerts on 429 versus 503. The example below shows both changes. Then remove labels that no dashboard or rule queries. If nobody reads a label, it only adds storage.
Next, check histograms. Each bucket, including +Inf, is its own series, and _sum and _count add two more. All of that is multiplied by every other label combination. As an illustrative estimate, 12 buckets across 40 routes, 3 status classes, and 20 pods comes to 14 × 40 × 3 × 20 = 33,600 series from one metric. Use fewer buckets, placed around your SLO thresholds.
To keep new problems out, add review rules for labels. Each new label should come with the query it serves and a stated, bounded set of values. Fixing a label when it is added costs a few minutes. Leaving it costs relabel rules to maintain, ongoing storage spend, and a slow TSDB during incidents.
This change gives up detail permanently. Questions like "which customer saw these errors" belong in logs and traces, or in exemplars that link a histogram observation to a trace ID.
import time
from flask import Flask, g, request
from prometheus_client import Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
app = Flask(__name__)
# Expose /metrics from the same process.
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {"/metrics": make_wsgi_app()})
REQUEST_LATENCY = Histogram(
"http_request_duration_seconds",
"HTTP request latency by route template and status class",
labelnames=["method", "route", "status_class"],
# Few buckets, placed around a 300ms SLO. Each bucket (plus +Inf, _sum, _count)
# is a separate series for every method/route/status_class combination.
buckets=(0.05, 0.1, 0.3, 0.5, 1.0, 2.5),
)
KNOWN_METHODS = {"GET", "POST", "PUT", "PATCH", "DELETE"}
@app.before_request
def start_timer():
g.start = time.perf_counter()
@app.after_request
def record_latency(response):
# url_rule.rule is the matched template, e.g. "/users/<int:user_id>",
# not the raw path "/users/48213". Unmatched paths share one value.
route = request.url_rule.rule if request.url_rule else "unmatched"
# 200, 201, 204 -> "2xx"; 429, 503 -> "4xx", "5xx".
status_class = f"{response.status_code // 100}xx"
# Clients can send arbitrary method strings; keep the label bounded.
method = request.method if request.method in KNOWN_METHODS else "other"
REQUEST_LATENCY.labels(method, route, status_class).observe(time.perf_counter() - g.start)
return response
@app.route("/users/<int:user_id>")
def get_user(user_id):
# user_id stays out of metric labels; put it in logs or trace attributes instead.
return {"id": user_id}
A Flask app instrumented with prometheus_client that labels request latency by the matched route template, status class, and a bounded method set, and uses a short bucket list, instead of raw paths, individual status codes, or user IDs.
How does Prometheus relabeling work as a containment tool?
Relabeling happens at two points in the scrape pipeline, and the difference between them determines what it can save you:
service discovery → relabel_configs (targets) → scrape → metric_relabel_configs (samples) → TSDB
relabel_configs runs on discovered targets before any request is made. It decides which targets to scrape and which target labels they get. metric_relabel_configs runs on every scraped sample before it reaches storage. This is the stage you use to contain series from third-party exporters whose code you cannot change.
| Action | Matches against | Typical use |
|---|---|---|
drop |
Joined source_labels values, often __name__ |
Discard a whole metric family, such as per-request histograms from a database exporter |
keep |
Joined source_labels values |
Allowlist only the metrics your dashboards and alerts use |
labeldrop |
Label names | Strip a label like request_id from every sample |
replace |
source_labels values, rewritten into target_label |
Collapse values, for example rewriting /users/8812 to /users/:id |
The cost limit is that metric_relabel_configs discards data after the scrape. The exporter still builds the full response, and Prometheus still pulls it over the network and parses every line before dropping anything. You reduce TSDB memory and storage, but network and scrape CPU costs stay the same. Fixing the instrumentation is still the better option. Use relabeling when you cannot change the source.
Give every rule an owner and a comment that explains what it removes and why. Undocumented rules spread across scrape configs become hidden logic. An instrumentation owner can add a label, see it missing in queries, and spend hours debugging an exporter that is working correctly.
scrape_configs:
- job_name: db-exporter
static_configs:
- targets: ['db-exporter.internal:9187']
metric_relabel_configs:
# owner: platform-db | Drop per-query histogram buckets; no dashboard uses them.
- source_labels: [__name__]
regex: 'db_query_duration_seconds_bucket'
action: drop
# owner: platform-db | request_id is unique per request and creates a series per value.
- regex: 'request_id'
action: labeldrop
# owner: api-team | Collapse numeric user IDs so each route is one series.
- source_labels: [path]
regex: '/users/[0-9]+'
target_label: path
replacement: '/users/:id'
action: replace
- job_name: queue-exporter
static_configs:
- targets: ['queue-exporter.internal:9419']
metric_relabel_configs:
# owner: messaging | Allowlist: keep only metrics referenced by alerts and dashboards.
# Anything not matched is discarded after the scrape, but it is still fetched and parsed.
- source_labels: [__name__]
regex: 'queue_depth|queue_consumers|queue_publish_total'
action: keep
Scrape config that uses metric_relabel_configs with drop, labeldrop, replace and keep to contain series from third-party exporters, with an owner and reason recorded on each rule.
When should you avoid labeldrop, and what are the safer alternatives?
labeldrop is safe only when the label carries no identity, meaning no two series differ solely by that label. If that does not hold, dropping it produces collisions. Take two series from one scrape:
http_requests_total{job="api", route="/users", status="200"}with value 140http_requests_total{job="api", route="/orders", status="200"}with value 95
After dropping route, both become http_requests_total{job="api", status="200"} with the same timestamp. Prometheus rejects one as a duplicate sample, which surfaces as ingestion errors in scrape metrics and logs. Whichever value survives is wrong: the true total is 235, but you get 140 or 95. The same mistake in remote-write relabeling can be worse. A backend may accept interleaved values from different sources as one series, and counters then appear to reset constantly. Rate calculations become meaningless, and nothing flags it.
Safer alternatives
The first option is to fix the exporter so the label is never emitted. Where you need the detail locally but not everywhere, aggregate it explicitly with a recording rule such as sum without (route) (rate(http_requests_total[5m])). Summing is correct where dropping is not, because it combines values instead of discarding all but one.
A common pattern is to keep raw series on the local Prometheus with short retention. You then send only the aggregated series to dashboards and long-term storage. Be explicit about the cost: after the raw data expires, you can no longer break down historical traffic by route.
Recording rules are not free. Each rule adds evaluation work and creates new series. If you group by the wrong dimensions, for example sum by (route, pod) when the raw metric had few pods per route, you can add series instead of removing them.
# rules/http_aggregation.yml (loaded via rule_files in prometheus.yml)
# Safe alternative to labeldrop: sum away the route label instead of
# discarding it, so values from different routes are added, not overwritten.
groups:
- name: http_aggregation
interval: 1m
rules:
# Keeps job, instance, status, etc. Only route is summed away.
- record: job_status:http_requests:rate5m
expr: sum without (route) (rate(http_requests_total[5m]))
---
# prometheus.yml (excerpt)
# Raw per-route series stay local with short retention, set on the CLI:
# --storage.tsdb.retention.time=7d
scrape_configs:
- job_name: api
static_configs:
- targets: ["api-1.internal.example:8080"]
# UNSAFE, do not do this: route distinguishes series, so dropping it
# makes /users and /orders collide as duplicate samples.
# metric_relabel_configs:
# - action: labeldrop
# regex: route
remote_write:
- url: ""
# Send only the aggregated series to long-term storage.
# Per-route history is lost once local raw data expires.
write_relabel_configs:
- source_labels: [__name__]
regex: "job_status:http_requests:rate5m"
action: keep
Shows a recording rule that sums away the route label, with raw per-route series kept locally under short retention and only the aggregated series sent over remote write, next to the commented-out labeldrop that would cause series collisions.
How do you prevent cardinality regressions?
Treat cardinality as a budget you enforce, not something you look at during an incident.
Set per-scrape limits
In each scrape job, set sample_limit to a ceiling somewhat above the target's normal sample count. Add label_limit, label_name_length_limit, and label_value_length_limit to catch payloads such as stack traces or full URLs stored as label values. Base each ceiling on what the job's targets actually expose, not on one global number.
These limits fail closed. If a target goes over sample_limit, Prometheus rejects the whole scrape, not just the extra series. The target's up drops to 0, and every metric from that target goes missing, including the ones your dashboards and SLO alerts depend on. A single bad label can therefore blind you to an entire service. Always pair limits with an alert on prometheus_target_scrapes_exceeded_sample_limit_total so a rejected scrape is reported as its own cause, not as a generic "target down."
Alert on growth before the ceiling
Alert on relative growth of prometheus_tsdb_head_series, for example by comparing the current value against the same metric with offset 1d. Separately, watch per-target scrape_samples_scraped and scrape_series_added. The per-target signals tell you which job introduced the new series.
Review in CI and dashboards
Keep a dashboard of the top metrics by series count. In code review, flag new labels whose values come from user input, IDs, or timestamps.
The payoff is timing. A reviewer who catches user_id in a diff spends minutes on it. Finding the same label during an outage, or on an invoice, can cost days of debugging and cleanup.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to prometheus high cardinality, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.