What is metrics cardinality and how is it counted?
In Prometheus, a time series is one metric name plus one unique combination of label values. http_requests_total{route="/login", status="200", method="POST", pod="api-7"} is one series. Change any single label value and you get a new series, with its own entry in the index and its own stream of samples. Cardinality is the number of these distinct series.
The worst case is the product of the distinct values of every label, because each label can in principle pair with every value of every other label. Illustrative estimate, with assumed inputs: 1 metric × 50 routes × 5 status codes × 4 methods × 200 pods = 200,000 series. Add a user_id label with 10,000 active users and the same metric could reach 2 billion series. A label with unbounded values multiplies the whole set rather than adding to it.
Histograms are usually where this hurts most. A classic histogram emits one series per bucket, plus _sum and _count, and every one of those is multiplied by the full label set. Removing a bucket or a label from a histogram therefore saves more series than the same change on a counter.
| Scenario (illustrative estimate) | Series per label combination | Worst-case series |
|---|---|---|
| Counter, labels above | 1 | 200,000 |
Histogram, 10 buckets incl. +Inf |
12 | 2,400,000 |
Histogram, drop pod label |
12 | 12,000 |
Counter, add user_id (10,000) |
1 | 2,000,000,000 |
Multiplication gives you an upper bound. Real counts depend on which combinations actually occur. A route that only accepts GET never produces POST series, for example. Use the products to judge risk, then measure the actual series count before deciding what to cut.
Why does high cardinality drive your monitoring bill?
On a self-hosted Prometheus server, an extra label costs memory, disk, and slower queries. You pay for those through hardware, so the cost stays hidden until the server struggles. Hosted backends turn the same series into a billing unit. Most charge per active time series, or per custom metric, which is usually defined as a unique combination of metric name and tag values. A label that only nudged your server's memory footprint becomes a line item on every invoice.
If you are evaluating a per-custom-metric model, map your Prometheus series count onto the vendor's billing unit before you migrate. Pay attention to histograms. In Prometheus, each bucket, plus _sum and _count, is a separate series. Find out how the vendor counts those, how it counts summaries, and whether it bills on peak, average, or distinct series across the billing period.
Churn makes this worse. Autoscaling and frequent deploys give every new pod new pod and instance values, which creates fresh series while the old ones go stale. Instantaneous cardinality can look modest while the number of distinct series seen over a month is much higher. Here is an illustrative estimate with assumed inputs: 2,000 live series per pod, 20 pods, and ten rolling deploys a day. Over a month, that could produce several million distinct series.
All of this is decided in instrumentation code and scrape configs. The engineers who write them should own the cost. Catching it at review time is cheaper than discovering it during an incident or a billing review.
Who should trim labels before billing?
This work pays off for the following groups:
- Teams shipping Prometheus data to a hosted backend that bills per active series or per custom metric. Every series you avoid sending is a series you do not pay for, so this group sees the most direct return.
- Teams running self-hosted Prometheus under memory pressure. The head block holds every active series in memory. Series count, more than sample count, decides how large the server has to be and how long it takes to restart and replay its write-ahead log.
- Platform or observability teams that own shared scrape configs. They can apply relabeling and limits centrally while application teams fix instrumentation at their own pace.
- Teams running autoscaled or frequently redeployed workloads on Kubernetes. Churn from
podandinstancelabels hits them hardest, so they gain the most from aggregating those labels away.
If your total series count is small, stable, and comfortably within your server's capacity or your plan's allowance, a full trimming project is probably not worth the effort. Adding the guardrails described below is still cheap insurance.
How does label trimming reduce series count?
Prometheus identifies a series by its complete label set, including __name__. Ingest hashes that set and uses the result to find an existing series in the head block or to create a new one. Any reduction in cardinality therefore comes from one of three mechanisms:
- Removing a label. Series that differed only by that label now have the same label set. Their count collapses by that label's number of distinct values. This is why the "drop
pod" row in the table above falls from 2,400,000 to 12,000. - Mapping many values to fewer values. Rewriting
/users/8812/ordersto/users/:id/orders, or503to5xx, shrinks one factor in the product without removing the dimension. - Not creating or not keeping the series at all. This covers dropping unused metrics, removing histogram buckets, or forwarding only pre-aggregated series.
You can apply these mechanisms at four points in the pipeline, each later than the last:
- Instrumentation code. The series is never exported.
metric_relabel_configs. The series is exported and scraped, then discarded or rewritten before storage.- Recording rules. Raw series are stored locally and aggregated into new, smaller series.
write_relabel_configsonremote_write. This decides which stored series leave the server and reach the paid backend.
The earlier the stage, the more cost it removes: exporter CPU, network, parsing, storage, and billing. The later the stage, the less coordination it needs, because you can change it without touching application code. The sections below cover each stage in that order.
One constraint applies everywhere. If two series end up with the same label set, Prometheus does not sum them. Collapsing labels at ingest or write time is only safe when the collapsed series were redundant. Where you need a correct total, aggregate with sum in a query or recording rule.
How do you find the labels that multiply your series count?
Start broad, then narrow. The query topk(10, count by (__name__)({__name__=~".+"})) ranks metric names by how many series each one currently holds. It touches every series in the head block, so it is expensive. Run it once against a replica or during a quiet period, not on a dashboard that refreshes every 30 seconds. The output usually shows a few metrics accounting for a disproportionate share of the total, and those are where to start.
For each suspect metric, check its labels one at a time. count(count by (label_name) (metric_name)) returns the number of distinct values that label takes on that metric. Multiply the per-label counts and compare the product to the metric's actual series count. If one label has thousands of values while the others have a handful, you have found the multiplier.
The Prometheus UI provides much of this without writing queries. Status → TSDB Status lists the top metric names by series count, the label names with the most distinct values, and the label-value pairs attached to the most series. For on-disk data, promtool tsdb analyze run against the data directory reports the same categories for a persisted block. This helps when the live server is too loaded to query.
All of these are snapshots, and snapshots miss churn. A label like a pod name or request ID can keep the instantaneous count modest while constantly creating new series that each cost indexing work and storage before going stale. Graph rate(prometheus_tsdb_head_series_created_total[5m]) next to prometheus_tsdb_head_series. A high creation rate with a flat total means series are being replaced as fast as they are created. That cost will not appear in any top-10 list.
#!/usr/bin/env bash
set -euo pipefail
PROM=""
METRIC="http_requests_total" # suspect metric from step 1
q() { # run an instant query, print "value<TAB>labels"
curl -s -G "$PROM/api/v1/query" --data-urlencode "query=$1" \
| jq -r '.data.result[] | "\(.value[1])\t\(.metric)"'
}
echo "== Top 10 metric names by series count (expensive, run once) =="
q 'topk(10, count by (__name__)({__name__=~".+"}))'
echo "== Distinct values per label on $METRIC =="
for label in method status path instance; do
printf '%s: ' "$label"
q "count(count by ($label) ($METRIC))" | cut -f1
done
printf 'actual series: '; q "count($METRIC)" | cut -f1
echo "== Same data the UI shows under Status -> TSDB Status =="
curl -s "$PROM/api/v1/status/tsdb" \
| jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"'
echo "== Churn: new series/sec vs. current head total =="
q 'rate(prometheus_tsdb_head_series_created_total[5m])'
q 'prometheus_tsdb_head_series'
A shell workflow that ranks metrics by series count, counts distinct values per label on a suspect metric, pulls the TSDB status data from the HTTP API, and compares the series creation rate with the head series total to expose churn.
Which Prometheus label patterns are unbounded?
A label is unbounded when its set of values grows with traffic, users or time rather than with your code. A quick test: could you list every value the label will ever take? If not, it does not belong on a metric.
| Pattern | Why it grows | Bounded replacement | Where the detail goes |
|---|---|---|---|
| User, session or request ID | One series per entity, forever | Drop it, or use a small enum such as plan tier | Logs and traces, linked through exemplars |
Raw URL path (/users/8812/orders) |
IDs embedded in the path | Route template (/users/:id/orders) |
Access logs |
| Error message string | Interpolated values make every message unique | Error class or code | Logs, span events |
| Full SQL or query text | Literals vary per call | Operation plus table name | Slow-query log, span attributes |
| Timestamp or build hash as a label value | New value every event or deploy | Put the timestamp in the sample value. Put the version on a single build_info gauge |
Deploy records |
| Pod or container name on short-lived workloads | New name on every restart or rollout | Deployment or workload name | Kubernetes events, logs |
The last row needs care because it looks bounded. At any moment you might have only 50 pods. Every rollout, autoscaling event or crash loop still mints a fresh set of series. The old series stop receiving samples but stay in the head block until it compacts, and they remain in queries across your retention window. This is the churn pattern from the previous section. Keep pod identity only on metrics you debug per instance, and aggregate it away everywhere else.
How do you fix cardinality at the source?
Fixing the instrumentation is the most effective option because series that are never created never have to be scraped, stored, or dropped later. Three changes cover most of the damage.
Label HTTP metrics with the route template. Record /users/:id as the router matched it, not the resolved /users/48213. Your code bounds the set of templates. Your data bounds the set of resolved paths, which means nothing bounds it. Most frameworks expose the matched route to middleware, so read it there rather than from the raw request URL.
Map errors to a fixed set of classes. Exception messages and upstream error strings embed IDs, hostnames, and timestamps. Map them to a small enum such as timeout, connection_refused, validation, upstream_5xx, and unknown, and write the full message to a log.
Trim histogram buckets. Every bucket is a separate series, multiplied by every other label combination. If your SLO thresholds are 100 ms and 500 ms, a fine spread of sub-millisecond buckets adds series that no query reads.
Put per-entity detail such as user, tenant, or order ID in logs, traces, or exemplars. Exemplars attach a trace ID to a sample without creating a new series. This relocates cost rather than removing it, because log and trace backends have their own volume pricing and retention limits.
Both kinds of removal have costs. A dropped label is gone for historical debugging, so you cannot re-slice last month's data by it. Stripping pod or instance labels also makes it harder to isolate the one misbehaving replica during an incident.
import logging, time
from flask import Flask, g, request
from prometheus_client import Counter, Histogram
from werkzeug.exceptions import HTTPException
app = Flask(__name__)
log = logging.getLogger("app")
LATENCY = Histogram("http_request_duration_seconds", "Request latency",
["method", "route", "status"],
buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5))
ERRORS = Counter("app_errors_total", "Errors by class", ["error_class"])
def classify(exc):
# Fixed enum: label values never come from exception messages.
if isinstance(exc, TimeoutError): return "timeout"
if isinstance(exc, ConnectionRefusedError): return "connection_refused"
if isinstance(exc, ValueError): return "validation"
return "unknown"
@app.before_request
def start_timer():
g.start = time.perf_counter()
@app.after_request
def record(response):
# url_rule.rule is the template ("/users/<int:user_id>"), not "/users/48213".
route = request.url_rule.rule if request.url_rule else "unmatched"
LATENCY.labels(request.method, route, str(response.status_code)).observe(
time.perf_counter() - g.start)
return response
@app.errorhandler(Exception)
def handle_error(exc):
if isinstance(exc, HTTPException):
return exc # 404s etc. are already counted via the status label
ERRORS.labels(classify(exc)).inc()
log.exception("request failed path=%s", request.path) # full detail goes to logs
return {"error": "internal"}, 500
Flask middleware that labels latency by the matched route template, maps exceptions to a fixed set of error classes with the full message sent to logs, and limits histogram buckets to values around the SLO thresholds.
How do you trim labels at scrape time when you can't change the code?
When you can't ship a code change, Prometheus can rewrite labels during ingestion. Prometheus has two relabeling stages. relabel_configs runs on discovered targets before the scrape. It decides which targets to scrape and which target labels they get, and it never sees individual series. metric_relabel_configs runs on every scraped sample after parsing and before storage, so it is the stage that controls series count.
Three actions do most of the work:
dropwithsource_labels: [__name__]and a regex discards whole metrics, such as an unused histogram's_bucketseries.labeldropremoves labels whose names match a regex.labelkeepdoes the inverse and removes every label that doesn't match. Uselabelkeepcarefully, because it also removesjobandinstanceunless the regex includes them.replacewith a capturing regex rewrites values into buckets. For example, it can turn/users/8812/ordersinto/users/:id/orders, or reduce status codes to2xxand5xx.
| Stage | Acts on | Reduces stored series | Reduces scrape/exporter cost |
|---|---|---|---|
relabel_configs |
Targets | Only by dropping whole targets | Yes, for dropped targets |
metric_relabel_configs |
Scraped samples | Yes | No |
| Fix in code | Exported metrics | Yes | Yes |
This approach does not save the scrape work. The target still builds and serializes every series, the network still carries them, and Prometheus still parses each sample before discarding it. Those costs remain until the code is fixed.
labeldrop has a second risk. If two series differ only by the dropped label, they become identical after the drop. Prometheus then rejects samples as duplicates or out-of-order, and the series it keeps holds one source's value, not the total. If you want a sum across a label, compute it with sum without(...) in a recording rule or query. Use labeldrop only for labels that never distinguish one series from another.
scrape_configs:
- job_name: checkout-api
static_configs:
- targets: ["checkout-api.internal:8080"]
# Runs on every scraped sample after parsing, before storage.
# The target still exposes and Prometheus still parses everything below.
metric_relabel_configs:
# drop: discard an unused histogram's bucket series entirely.
- source_labels: [__name__]
regex: "http_request_duration_seconds_bucket"
action: drop
# labeldrop: strip labels by name (regex matches label names).
# Only drop labels that are constant per target, so no two series
# become identical. Dropping a per-request label such as request_id
# would merge series and cause duplicate-sample errors.
- regex: "git_commit|owner_team"
action: labeldrop
# replace: collapse numeric user IDs in the path into one value.
# Prometheus anchors the regex, so it must match the whole value.
- source_labels: [path]
regex: "/users/[0-9]+/(.*)"
target_label: path
replacement: "/users/:id/${1}"
action: replace
# replace: reduce status codes to classes (200 -> 2xx, 503 -> 5xx).
# ${1} needs braces; $1xx would be read as a group named "1xx".
- source_labels: [status]
regex: "([1-5])[0-9][0-9]"
target_label: status
replacement: "${1}xx"
action: replace
# Both replace rules map many inputs to one output, so they can
# produce colliding series just like labeldrop does. Check for
# duplicate or out-of-order sample errors after deploying them,
# and use recording rules where you need correct totals.
A scrape job whose metric_relabel_configs drops an unused histogram's buckets, strips labels that are constant per target, and rewrites high-cardinality path and status values into buckets, with comments noting where each rule can merge series.
How do you aggregate before shipping and keep cardinality from coming back?
Keep full-resolution data in Prometheus with short local retention, and send only pre-aggregated series to paid storage:
app ──scrape──► Prometheus
├─ metric_relabel_configs (drop/trim on ingest)
├─ recording rules (sum by the dimensions dashboards query)
├─ local TSDB, short retention (raw data for debugging)
└─ remote_write + write_relabel_configs
(keep only recorded series) ──► vendor
Recording rules compute the aggregates, for example request rate by service and status_class. write_relabel_configs on remote_write then keeps only those recorded series, so the vendor never receives the per-pod raw data. This has costs. Rule evaluation adds CPU and memory load to Prometheus. The recorded series are a second set of names you have to maintain. On the paid backend, ad hoc queries can only use the dimensions you chose in advance. Anything else means querying local Prometheus before its retention window expires.
Guardrails
Set sample_limit and label_limit on each scrape job. If a target exceeds sample_limit, Prometheus rejects the entire scrape, not just the extra series, so that target's metrics go missing. Set the limit well above normal volume and alert on prometheus_target_scrapes_exceeded_sample_limit_total. Also alert on prometheus_tsdb_head_series (total active series) and on the rate of prometheus_tsdb_head_series_created_total. A spike in new series creation usually means a deploy introduced an unbounded label.
Review process
Repeat these steps on a regular schedule:
- List metrics that no dashboard or alert queries, and drop them.
- Agree a team convention for allowed label sets and a maximum number of distinct values per label.
- Add a cardinality check to code review for new instrumentation.
Catching a bad label in review costs one comment. Finding it on the invoice means a cleanup project.
rule_files:
- rules/aggregates.yml
scrape_configs:
- job_name: api
sample_limit: 20000 # well above normal volume: exceeding it fails the WHOLE scrape
label_limit: 30 # reject targets that emit series with too many labels
static_configs:
- targets: ["api-1.internal:8080", "api-2.internal:8080"]
metric_relabel_configs:
- source_labels: [__name__] # drop series no dashboard or alert uses
regex: "go_gc_duration_seconds.*"
action: drop
- regex: "request_id|user_id" # strip unbounded labels on ingest
action: labeldrop
remote_write:
- url:
write_relabel_configs:
- source_labels: [__name__] # ship only recorded series (level:metric:ops naming)
regex: ".+:.+"
action: keep
---
groups:
- name: aggregates
interval: 1m
rules:
# Pre-aggregate to the dimensions dashboards query; per-pod detail stays local.
- record: service_status_class:http_requests:rate5m
expr: |
sum by (service, status_class) (
label_replace(rate(http_requests_total[5m]),
"status_class", "${1}xx", "code", "([0-9])..")
)
A Prometheus config that trims labels on ingest, caps each scrape with sample_limit and label_limit, records aggregates by service and status class, and uses write_relabel_configs so remote_write ships only the recorded series.
When should you avoid trimming labels?
Trimming is a trade of detail for cost. In some situations the trade is a bad one:
- The label is what you alert or report on. If you have per-tenant SLOs, contractual per-customer reporting, or alerts that must name the failing instance, dropping
tenantorinstanceremoves the signal you need. Keep the label and bound it instead, for example by tracking only named top-tier tenants and grouping the rest asother. - You are in the middle of an incident or an open investigation. Removing labels while people are still slicing data by them destroys evidence. Wait until the investigation is closed.
- The cardinality is already bounded and affordable. A label with a dozen stable values costs little. Cutting it adds config to maintain and can merge series for no meaningful saving.
- Your billing is not driven by series count. Some backends price on ingested samples, bytes, or query volume. In that case, scrape interval, retention, and query patterns can matter more than label count, so check the pricing model before you start trimming.
- You cannot verify the effect.
labeldropand many-to-onereplacerules can silently merge series and corrupt totals. If you have no way to check for duplicate-sample errors or compare totals before and after, do not apply them to metrics that feed billing, capacity, or SLO decisions. - The question is per-entity by nature. "Which user saw this error?" is a question for logs or traces. Trimming a metric does not answer it, and keeping
user_idon the metric to answer it is the wrong fix. Move the question to the right tool rather than tuning the metric.
What are the alternatives to trimming labels?
Trimming is not the only way to control cardinality cost. Other options either reduce cost through a different mechanism or move the data somewhere that prices it differently:
- Native histograms. Prometheus native histograms store the bucket distribution inside a single series instead of one series per bucket. This removes the per-bucket multiplier described above. Check whether your Prometheus version, client library, and remote backend support them before relying on them, and how your backend bills for them.
- Logs and traces for high-cardinality detail. Request IDs, user IDs, and raw paths belong in structured logs or trace attributes. Those backends are designed for per-event data and are queried differently. They also have their own volume costs, so this moves cost rather than removing it.
- Exemplars. Exemplars attach a trace ID to individual samples, so you can jump from a latency spike to a specific request without adding a label.
- Aggregation in the pipeline outside Prometheus. A collector or proxy between your applications and the backend can aggregate or drop dimensions in transit. This centralizes the policy, but it adds another component to operate and the same series-merging risks apply.
- Self-hosted long-term storage. Running your own horizontally scalable Prometheus-compatible store turns a per-series bill into an infrastructure and operations cost. High cardinality still costs memory and query performance, so this changes who pays rather than whether you pay.
- Sharding or federation. Splitting scrape load across several Prometheus servers spreads memory pressure, and federating only aggregated series upward limits what reaches the central server. This helps self-hosted capacity, but it does not reduce what a per-series vendor bills unless you also limit what is forwarded.
- Longer scrape intervals or shorter retention. These reduce sample volume and storage, not series count. They help when cost is driven by samples or bytes, and do little under per-active-series billing.
- Accepting the cost. If a high-cardinality metric answers a question that matters and no cheaper tool answers it as well, paying for it can be the right decision. Make it a deliberate decision with a named owner, not an accident discovered on an invoice.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to metrics cardinality cost, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.