What is Prometheus high cardinality and why does it multiply?
In Prometheus, a time series is identified by its metric name plus its complete set of label key-value pairs. http_requests_total{method="GET", route="/orders/:id", status="2xx"} and the same metric with status="5xx" are two separate series. Each one gets its own entry in the in-memory head block, which holds recent samples along with index and label data for every active series. Memory use therefore grows with the number of distinct series.
This is why cardinality multiplies. The upper bound on series for a metric is the product of the distinct values of each label.
Illustrative estimate (assumed inputs, not measured):
method: 5 valuesroute(templated, e.g./orders/:id): 40 valuesstatus(class, e.g.2xx): 5 values
5 × 40 × 5 = 1,000 possible series per scrape target. Across 20 instances, that is up to 20,000. Not every combination will occur, but this is the ceiling you plan capacity against.
Now add user_id with an assumed 10,000 active users. The ceiling becomes 1,000 × 10,000 = 10 million series per instance. One unbounded label turns a manageable metric into one that can exhaust the server's memory.
Churn is a second source of growth. Labels such as pod or instance take new values every time a pod is rescheduled or redeployed. The old series stop receiving samples, but they stay in the head block and index until they age out. A cluster that redeploys often can carry several generations of series simultaneously, even if its steady-state cardinality looks modest.
METRIC: http_requests_total (illustrative estimate, assumed inputs)
+-----------+ +-------------+ +-------------+
| method | | route | | status |
| 5 values | x | 40 values | x | 5 values | = 1,000 series/target
+-----------+ +-------------+ +-------------+ x 20 instances = 20,000
|
v add user_id (10,000 values)
+----------------------------------+
| 1,000 x 10,000 = 10,000,000 |
| series ceiling per instance |
+----------------------------------+
|
v each unique label set = 1 series
+-----------------------------------------------------------------+
| PROMETHEUS HEAD BLOCK (in memory) |
| {method="GET",route="/orders/:id",status="2xx"} -> samples |
| {method="GET",route="/orders/:id",status="5xx"} -> samples |
| ... + index and label data for every active series |
+-----------------------------------------------------------------+
^
| churn: new pod label value per redeploy
+--------------+ +--------------+ +--------------+
| pod=api-a1 |-->| pod=api-b7 |-->| pod=api-c3 |
| stale, held | | stale, held | | active |
| until age-out| | until age-out| | |
+--------------+ +--------------+ +--------------+
Label value counts multiply into the series ceiling, so a single unbounded label like user_id inflates it 10,000-fold, while pod churn leaves stale series in the head block until they age out (illustrative estimate, assumed inputs).
Which labels cause high label cardinality?
The question to ask of any label is whether you could write down every value it will ever take. If you can, it is bounded. If the set grows with traffic, users or time, it is unbounded, and every new value creates a new series for every metric that carries the label.
| Unbounded (avoid as labels) | Bounded alternative |
|---|---|
| User ID, email, account ID | None; put it in logs or traces |
| Request ID, trace ID | Exemplars or traces |
Raw URL path (/orders/8812) |
Route template (/orders/{id}) |
| Client or pod IP address | Zone, node pool, or nothing |
| Error message string | Error class or error code |
| Status code from arbitrary upstreams | Status class (2xx, 4xx, 5xx) |
| Tenant ID, container ID in a growing fleet | Tenant tier, workload name |
Raw paths and error strings are the most common surprises. They look harmless in a test environment with ten requests, but in production a crawler hitting random URLs or a parser embedding input in its error message can generate thousands of values in an hour.
Some labels seem bounded and still churn. Pod names, container IDs and instance addresses are finite at any moment, but each rollout replaces them. A deployment with 50 pods that ships twice a day produces 100 new values daily, each carrying its own copy of every series. This churn is the multi-generation effect described above.
Anyone adding instrumentation needs to apply this test, because the person who adds the label rarely pays the cost. On a shared Prometheus server, one team's careless label slows queries and consumes memory for everyone else, so platform owners should review label choices too.
How do you find high-cardinality metrics in Prometheus?
Prometheus records enough about its own data to find offenders without any external tooling. Start with a ranking by metric name: topk(10, count by (__name__)({__name__=~".+"})) returns the metrics that own the most series. This query touches every series in the head block, so run it sparingly on a large server, and prefer an instant query over a range query.
Once you have a suspect metric, find out which label is driving it. count(count by (path)(http_requests_total)) gives the number of distinct path values. Repeat the query for each label on the metric. The label whose distinct count tracks requests, users or pods rather than a small fixed set is usually the problem.
The TSDB status page in the web UI (Status → TSDB Status) shows the same picture without writing queries. It lists the metric names with the most series, the label names with the most distinct values, and the label pairs that use the most memory. It reads from the head block only, so it reflects recent data, not total history.
Cardinality also changes over time, and a single snapshot can't show that. Graph prometheus_tsdb_head_series alongside process_resident_memory_bytes to catch steady growth. Also graph rate(prometheus_tsdb_head_series_created_total[5m]) to catch churn: series that appear and disappear quickly, for example on every pod restart. Churn inflates memory and the index even when the live series count looks flat.
When to run these checks:
- Routinely, before capacity problems appear. Growth is cheap to fix while it is gradual.
- After every deployment, when new labels first show up.
- During incident review, whenever Prometheus itself was slow or ran out of memory.
#!/usr/bin/env bash
# Cardinality check against a Prometheus server using only its HTTP API.
# Run routinely, after deployments, and during incident review.
set -euo pipefail
PROM="${PROM:-"
# Helper: run an instant query (cheaper than a range query) and print metric + value.
q() {
curl -sG "$PROM/api/v1/query" --data-urlencode "query=$1" \
| jq -r '.data.result[] | "\(.metric | tostring)\t\(.value[1])"'
}
echo "== Top 10 metrics by series count (touches every head series; run sparingly) =="
q 'topk(10, count by (__name__)({__name__=~".+"}))'
echo "== Distinct values of the 'path' label on a suspect metric =="
q 'count(count by (path)(http_requests_total))'
echo "== Live head series, memory, and churn (series created per second) =="
q 'prometheus_tsdb_head_series'
q 'process_resident_memory_bytes{job="prometheus"}'
q 'rate(prometheus_tsdb_head_series_created_total[5m])'
# Same data as Status -> TSDB Status in the web UI (head block only).
echo "== TSDB status: top metric names and label names =="
curl -s "$PROM/api/v1/status/tsdb" | jq '{
headSeries: .data.headStats.numSeries,
topMetrics: .data.seriesCountByMetricName,
topLabelsByValueCount: .data.labelValueCountByLabelName,
topLabelsByMemory: .data.memoryInBytesByLabelName
}'
A bash script that queries Prometheus's own HTTP API to rank metrics by series count, count distinct values of a suspect label, check head series, memory and churn, and pull the same data the TSDB Status page shows.
How should you fix cardinality at instrumentation time?
The most effective fix happens before a sample is ever scraped. Relabeling can drop a bad label, but it cannot repair a metric whose design is wrong. That makes instrumentation code the place to fix cardinality.
- Label routes, not paths. Record the router's template, such as
/users/{id}, instead of/users/48213. The template set is finite and known at build time. Raw paths grow with traffic. - Bucket status codes.
2xx,4xxand5xxusually answer the alerting question. Keep exact codes only where a specific one, such as 429, drives a decision. - Move per-entity detail out of metrics. User IDs, order IDs and tenant keys belong in logs or trace attributes. Those systems are built for that kind of lookup.
- Treat each new label as a design decision. Write down its maximum value count in the review, and reject labels whose bound is "unknown."
Histograms need extra care. Every bucket is its own series, plus _sum and _count, so each label multiplies by the bucket count.
Illustrative estimate (assumed numbers, not a benchmark): series per target = routes × methods × status values × (buckets + 2)
- 40 × 4 × 5 × (12 + 2) = 11,200
- With 20 exact status codes instead of 5 classes: 44,800
- With buckets trimmed to 8: 8,000
Keep histogram label sets narrower than counter label sets, and drop buckets that no SLO or dashboard reads.
This approach has a development cost. Service owners have to agree on label conventions, and every fix needs a redeploy. In return, it removes a recurring class of incidents, which saves more engineering time than repeated firefighting.
import logging
import time
from flask import Flask, g, request
from prometheus_client import Histogram, start_http_server
app = Flask(__name__)
log = logging.getLogger("requests")
# Narrow, bounded label set, written down at review time:
# route ~40 templates x method ~4 x status_class 5 values.
REQUEST_LATENCY = Histogram(
"http_request_duration_seconds",
"HTTP request latency by route template",
["route", "method", "status_class"],
# Only the buckets the 300ms SLO and dashboards read; +Inf is added automatically.
buckets=(0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5),
)
@app.before_request
def start_timer():
g.start = time.perf_counter()
@app.after_request
def record(response):
# url_rule.rule is the template ("/users/<int:user_id>"), not the raw path.
# Unmatched requests share one fixed value instead of creating new series.
route = request.url_rule.rule if request.url_rule else "unmatched"
status_class = f"{response.status_code // 100}xx" # 2xx, 4xx, 5xx
REQUEST_LATENCY.labels(route, request.method, status_class).observe(
time.perf_counter() - g.start
)
# Per-entity detail (raw path, exact code) goes to logs, not labels.
log.info("path=%s status=%d", request.path, response.status_code)
return response
@app.route("/users/<int:user_id>")
def get_user(user_id):
return {"id": user_id}
if __name__ == "__main__":
start_http_server(8000) # expose /metrics on a separate port
app.run(port=8080)
A Flask service records request latency with the route template and status class as labels and a trimmed bucket list, and sends raw paths and exact status codes to logs instead of turning them into labels.
How does relabeling reduce cardinality at scrape time?
Relabeling is the fallback when a fix in code is slow or the code belongs to someone else, such as a third-party exporter. Prometheus has two relabeling stages, and only one of them sees metric labels.
| Stage | Config block | Operates on | Runs | Typical cardinality use |
|---|---|---|---|---|
| Target relabeling | relabel_configs |
Discovered targets and their metadata labels | Before the scrape | Skip targets entirely, control target-level labels |
| Metric relabeling | metric_relabel_configs |
Every scraped sample | After the scrape, before storage | Drop series, drop labels, bound label values |
Most cardinality work happens in metric_relabel_configs. It supports three main patterns:
- Drop whole series. Use the
dropaction with a regex on__name__, for example to remove a histogram's per-bucket series you never query. - Drop or replace a label. Use
labeldropto remove a label such asrequest_id. Usereplaceto overwrite it with a constant. - Bound the values. Use a
replacerule with a regex to collapse values. For example, rewrite/users/8812/ordersinto/users/:id/orders, or map any status code to2xx,4xxor5xx.
The trade-offs are real:
- Dropped detail is gone permanently. You cannot recover it later.
- Scrape cost stays the same. The exporter still generates the full payload, and Prometheus still transfers and parses it. Only storage and memory benefit.
- Dropping a label does not aggregate. If two series differ only by the dropped label, they collapse to identical label sets in the same scrape. Prometheus does not sum them. It ingests one sample, rejects the others as duplicates and flags the scrape with errors.
- Central rules drift from the code. When a developer renames a label or adds a route pattern, the regex silently stops matching. Keep these rules in version control next to the service and test them.
scrape_configs:
- job_name: checkout-api
static_configs:
- targets: ['checkout-api.internal:9100']
# Runs after the scrape, before storage. The exporter still emits
# and Prometheus still parses the full payload; only TSDB load shrinks.
metric_relabel_configs:
# 1. Drop whole series: per-bucket histogram series nobody queries.
# Regexes are fully anchored, so this matches the exact name only.
- source_labels: [__name__]
regex: 'http_request_duration_seconds_bucket'
action: drop
# 2. Remove an unbounded label. labeldrop matches label NAMES.
# If two series differ only by request_id, they collide in the
# same scrape and the extra samples are rejected as duplicates.
# They are not summed.
- regex: 'request_id'
action: labeldrop
# 3a. Bound values: collapse numeric IDs in the path label.
# If the regex does not match, the label is left unchanged.
- source_labels: [path]
regex: '/users/[0-9]+/orders'
target_label: path
replacement: '/users/:id/orders'
action: replace
# 3b. Bound values: map status codes to classes such as 2xx and 5xx.
- source_labels: [code]
regex: '([1-5])[0-9][0-9]'
target_label: code
replacement: '${1}xx'
action: replace
# Replace a label with a constant. With no source_labels, the
# default regex (.*) matches the empty string, so this always applies.
- target_label: pod_ip
replacement: 'redacted'
action: replace
A metric_relabel_configs block that drops an unused histogram series, removes an unbounded label, collapses path and status-code values into bounded sets, and overwrites a label with a constant, with comments marking where series collisions and unchanged scrape cost still apply.
When should you use sample limits, label limits and series-growth alerts?
Relabeling handles the problems you know about. Limits protect you from the ones you don't, such as a new deploy that starts emitting a user ID as a label. They are guardrails, not a fix.
Per-scrape limits
In a scrape config, sample_limit caps how many samples a target may return after metric relabeling. label_limit, label_name_length_limit and label_value_length_limit reject series whose label sets are too large or too long. When a target breaks one of these limits, Prometheus fails that scrape and ingests nothing from it. The target's up series goes to 0, and the rest of the server keeps working. The cost of a cardinality explosion stays with the job that caused it, so it does not exhaust the server's memory and take every other target with it.
The downside is that the failure is all-or-nothing. A target that goes one sample over its limit disappears entirely, often at the moment it is misbehaving and you most need its metrics. Set limits from observed baselines. Look at scrape_samples_post_metric_relabeling per job over a period that includes peak traffic and recent deploys, then add enough headroom for normal growth. Doubling the baseline is a reasonable starting point. Alert on up == 0 and on prometheus_target_scrapes_exceeded_sample_limit_total so that a tripped limit pages someone instead of failing silently.
Series-growth alerts
Limits work per target. To catch slow growth across many targets, alert on prometheus_tsdb_head_series. Use one alert for an absolute ceiling based on the server's memory, and another for an unusual rate of increase over a few hours. The rate alert catches a gradual leak early, while you can still fix it with relabeling before you have to raise limits.
What are the alternatives: recording rules, logs and traces?
Not every cardinality problem has to be solved by deleting labels. Two other options move the cost somewhere else.
Recording rules precompute aggregates on a schedule and store the result as new, smaller series. For example, a rule that sums request rate by service and status code drops the per-pod and per-route dimensions. Dashboards then read a few dozen series instead of fanning out across thousands, so queries get faster and cheaper. The raw series are still scraped, indexed and held in memory, so ingestion cost does not change. Rules also add evaluation work every interval and create their own output series. The biggest risk is that they hide the problem. Dashboards stay fast while the raw metric keeps growing underneath, so the limits and growth alerts from the previous section still need to cover the source metrics.
Logs and traces are the right home for per-entity detail such as user IDs, request IDs or full URLs. They handle unbounded values without a series index. The cost moves to log storage and trace backends, and the queries change too: you search and filter events instead of running cheap numeric aggregations. Trace sampling also means you lose completeness.
| Approach | Where it acts | What it costs | What it saves |
|---|---|---|---|
| Instrumentation fix | Application code | Code changes, deploys | Ingestion, storage, query |
| Relabeling | Scrape time | Config upkeep, lost detail | Ingestion, storage |
| Sample/label limits | Scrape time | Failed scrapes when tripped | Protects server memory |
| Recording rules | After ingestion | Evaluation load, extra series | Query cost only |
| Logs/traces | Other systems | Their storage, harder queries | Prometheus series entirely |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to prometheus high cardinality, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.