Prometheus High Cardinality Cost: Trimming Labels That Inflate Bills

What is metric cardinality and how does it multiply Prometheus series?

In Prometheus, a time series is a metric name plus one exact set of label key/value pairs. http_requests_total{method="GET", status="200"} and http_requests_total{method="GET", status="500"} are two separate series. Each one has its own index entry, its own in-memory chunk, and its own stream of samples. Cardinality is the number of distinct series that exist.

For a single metric, the maximum series count is the product of the distinct values of each label. Not every combination appears in practice, but the product sets the ceiling, and labels multiply rather than add.

Illustrative estimate. Assume a request-duration histogram with 40 endpoints, 5 status codes, 10 buckets (each bucket is its own series through the le label), and 200 pods. These inputs are assumptions, not measurements:

The pod label alone takes the total from 2,000 to 400,000. A user ID label with thousands of values would push it far higher. When you audit cardinality, look first for the one or two labels with large value sets. They usually account for most of the total.

Churn makes this worse. Labels such as pod name, container ID, or build hash take new values on every deploy or reschedule. Each new value creates a fresh set of series and leaves the old ones stale. A scrape snapshot might show 400,000 active series. Over a retention window that spans many rollouts, though, storage and the index hold several generations of them. Queries over long ranges then have to read every one of those generations.

Why does high cardinality drive Prometheus and managed Prometheus pricing?

In self-hosted Prometheus, the cost of that sprawl shows up as memory and disk. Every active series lives in the in-memory head block with its own label set and index entries, so RAM grows roughly in line with active series. Churned series stay in the on-disk index until retention expires. Sample volume matters less here because compression keeps per-sample storage small. The series count is usually what forces a bigger node or a sharding project.

Managed Prometheus services move that cost onto an invoice, and providers do not all measure it the same way:

Check your provider's billing dimension before you change anything, because it decides which fix pays off. As an illustrative estimate, 100,000 series scraped every 30 seconds produce about 8.6 billion samples over a 30-day month. Changing the interval to 60 seconds halves a sample-based bill without removing a single series. Under series-based pricing, the same change saves nothing. Dropping a high-cardinality label works the other way: it cuts a series-based bill directly, but on a sample-based bill it helps only in proportion to the samples those series produced.

This matters most for teams running multi-tenant services, large or autoscaling pod fleets, or anyone whose metrics bill is growing faster than their traffic.

Who should use label trimming, and who can skip it?

Label trimming is worth the effort when series count, not traffic, is what drives your cost or your capacity problems. Typical signs:

• Teams on managed Prometheus with per-series or per-sample billing. Every series removed before remote_write comes straight off the invoice.
• Operators of self-hosted Prometheus servers that are running short of memory. Head-block RAM tracks active series. Trimming labels can postpone or avoid a move to a larger node or to sharding.
• Platform teams running Kubernetes with frequent deploys or autoscaling. Pod, container and build-hash labels churn, and old generations of series keep consuming index space and query time until retention expires.
• Multi-tenant SaaS services. Tenant, customer or user labels grow with the business, not with the infrastructure.
• Teams whose dashboards or long-range queries are getting slow. Queries over many weeks of churned series read every generation.

You can usually skip a dedicated cleanup if a small, stable fleet produces a series count that fits comfortably in one server's memory and your bill is flat. Even then, the review checklist and scrape limits described later are cheap insurance against a single bad label shipping in a future release.

Ownership tends to split two ways. Application developers fix instrumentation, because they own the code that creates labels. Platform or observability engineers own relabeling, recording rules, remote_write filtering and limits, because those live in the Prometheus configuration.

How does label trimming work in the Prometheus pipeline?

A sample passes through several stages between your code and a paid backend. Each stage can remove labels or series, and each one affects only the stages after it:

1. Instrumentation. The client library in your application keeps one in-memory child for every label combination it has observed. Labels that are never set here never exist anywhere downstream.
2. Exposition and scrape. Prometheus fetches the /metrics endpoint at the scrape interval. The full set of exposed series crosses the network, whatever happens to them later.
3. Target relabeling (relabel_configs). Runs before the scrape on target metadata from service discovery. It decides which targets are scraped and which target labels, such as instance or job, are attached. It cannot see individual metric labels.
4. Metric relabeling (metric_relabel_configs). Runs on each scraped series before it is written. It can rewrite label values, drop label names, or drop whole series. Scrape limits such as sample_limit are evaluated against the series that remain after this step.
5. TSDB head and blocks. Surviving series enter the in-memory head block, then are compacted into on-disk blocks that stay until retention expires. This is where self-hosted memory and disk cost accumulates.
6. Recording rules. Evaluated on a schedule against stored data. They write new, usually lower-cardinality series, such as per-service rates summed over pods.
7. Remote write (write_relabel_configs). Filters and relabels series just before they are sent to a remote backend. This is where a managed service starts counting, so it is the last point at which you can avoid a series-based or sample-based charge.

The principle behind every fix in this article follows from that order. A label removed at stage 1 saves cost at every later stage. A label removed at stage 7 saves only the remote bill, while the local server and the application still carry it. The cardinality maths doesn't change at any stage: removing a label with N values divides the ceiling for that metric by up to N, provided the remaining labels still keep each series unique.

How do you find which metrics and labels are driving series count?

Start with the TSDB status page in the Prometheus web UI (under the Status menu). It reports head-block statistics: the metric names with the most series, the label names with the most distinct values, and the label-value pairs that appear in the most series. Prometheus has already computed these, so the page costs nothing to open, and it usually points straight at the worst offender.

To confirm what you see there, or to dig deeper, use PromQL. topk(20, count by (__name__) ({__name__=~".+"})) ranks metrics by series count. To find out which label is multiplying a given metric, count its distinct values with count(count by (some_label) (some_metric)). A label with thousands of values on a metric that has a few dozen series per pod is your target.

These queries touch every active series. On a large server they can take a long time, use a lot of memory, or hit query limits. Add a job matcher so they cover one job at a time, and run them off-peak.

For a per-target view, Prometheus records scrape_samples_scraped and scrape_series_added for every scrape. The first shows which targets expose the most samples. The second catches churn: a target that keeps adding new series on every scrape is minting label values, often from IDs or timestamps.

Before changing anything, record a baseline in the unit you are actually billed on. That is usually samples ingested or active series, as reported by your provider. Without that number you can't prove the cleanup saved money.

Copy#!/usr/bin/env bash
set -euo pipefail
PROM=""
JOB="api-server"   # scope expensive queries to a single job
OUT="baseline-$(date +%Y%m%d-%H%M).txt"

q() {  # run an instant query and print "labels value" lines
  curl -sG "$PROM/api/v1/query" --data-urlencode "query=$1" \
    | jq -r '.data.result[] | "\(.metric | tostring) \(.value[1])"'
}

{
  echo "## TSDB status: top metrics by series, top labels by distinct values"
  curl -s "$PROM/api/v1/status/tsdb" | jq -r '
    (.data.seriesCountByMetricName[] | "series  \(.value)\t\(.name)"),
    (.data.labelValueCountByLabelName[] | "values  \(.value)\t\(.name)")'

  echo "## Metrics ranked by series count (job=$JOB)"
  q "topk(20, count by (__name__) ({job=\"$JOB\"}))"

  echo "## Distinct values of a suspect label on a suspect metric"
  q "count(count by (path) (http_requests_total{job=\"$JOB\"}))"

  echo "## Targets exposing the most samples per scrape"
  q "topk(10, scrape_samples_scraped{job=\"$JOB\"})"

  echo "## Targets minting new series (churn) over the last hour"
  q "topk(10, sum_over_time(scrape_series_added{job=\"$JOB\"}[1h]))"

  echo "## Local baseline: active head series and samples/sec ingested"
  q "prometheus_tsdb_head_series"
  q "rate(prometheus_tsdb_head_samples_appended_total[5m])"
  # Also record the billed figure (samples ingested or active series) from your provider here.
} | tee "$OUT"

A script that pulls the TSDB status endpoint, runs job-scoped PromQL to rank metrics, label cardinality, and per-target samples and churn, and saves a timestamped baseline to compare against after trimming labels.

Which labels are high-risk and when should you keep them anyway?

High-risk labels fall into two groups. Unbounded labels have no ceiling on distinct values, so series count grows with traffic or users rather than with your infrastructure. Churny labels have a modest number of values at any moment but keep replacing them. Each replacement creates a new series, and the old one still counts toward storage and often toward billing until it ages out.

Most unbounded labels belong in logs or traces, where per-event detail is cheap to store. Replace raw paths with route templates such as /users/:id, and replace error messages with a small set of error classes. Email addresses and session tokens are also a data-exposure risk in a system rarely treated as sensitive.

Churny labels are harder to give up. A pod name is often the only way to find the one bad replica that is dragging down p99 latency. A common compromise is to keep pod-level labels on a few key metrics, such as request rate, errors and latency, and aggregate them away everywhere else.

Before dropping anything, search dashboards, alert rules, recording rules and SLO definitions for the label. A dropped dimension cannot be recovered for data already ingested without it.

How do you fix cardinality at instrumentation, ingestion and aggregation layers?

Cardinality can be cut at four points between the application and the paid backend. The earlier the fix, the less it costs, because every later layer still pays for the series it receives.

At the source

Fixing instrumentation is the cheapest option because the series never exist. Concretely:

• Record the route template (/users/{id}), not the raw path.
• Map errors to a fixed set of classes such as timeout, client_error and upstream_5xx instead of using raw messages.
• Move per-request detail like user IDs and trace IDs to logs or traces.
• Add a tenant label only when the tenant set is small and bounded.

At ingestion

metric_relabel_configs runs after the scrape and before samples are written to the TSDB. Its actions:

• labeldrop and labelkeep remove label names.
• drop discards entire series that match a regex.
• replace rewrites values, for example collapsing numeric IDs into a placeholder.

Do not confuse it with relabel_configs, which runs at target discovery before the scrape. It sees target metadata, not the labels on individual metrics.

labeldrop is only safe if the remaining labels still uniquely identify each series. If two series collapse into the same label set, their samples collide and Prometheus rejects them as duplicates. The data is not merged.

At aggregation

Recording rules precompute the aggregates your dashboards actually query, such as per-service request rate summed over pods. On remote_write, add write_relabel_configs so that only those recorded series, and any raw series you deliberately need, reach the billed backend. The local Prometheus keeps full detail for short-retention debugging, and you pay long-term storage only for the series that dashboards and alerts use.

Copyrule_files:
  - rules/aggregates.yml   # defines e.g. service:http_requests:rate5m

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ['api-1:9100', 'api-2:9100']
    # Ingestion layer: runs after the scrape, before samples hit the TSDB.
    metric_relabel_configs:
      # Collapse raw numeric IDs into the route template.
      - source_labels: [path]
        regex: '/users/[0-9]+'
        target_label: path
        replacement: '/users/{id}'
        action: replace
      # Drop a per-request label. Safe only if the remaining labels
      # still uniquely identify each series. Otherwise samples collide.
      - regex: request_id
        action: labeldrop
      # Discard whole series nobody queries.
      - source_labels: [__name__]
        regex: 'debug_.*'
        action: drop

remote_write:
  - url:
    # Aggregation layer: ship only recorded series (level:metric:op naming)
    # plus raw series you deliberately keep. Local TSDB retains full detail.
    write_relabel_configs:
      - source_labels: [__name__]
        regex: '.+:.+|up'
        action: keep

A prometheus.yml excerpt that rewrites, drops and filters labels in metric_relabel_configs at ingestion, then uses write_relabel_configs so that only recording-rule aggregates and up reach the paid remote_write backend.

How do you reduce histogram cardinality cost?

Every classic histogram produces one series per bucket (the _bucket series with an le label, including +Inf), plus _sum and _count. That whole set is repeated for every combination of the histogram's other labels. Buckets act as a multiplier inside a multiplier, so histograms are usually the first place to look once you know which metrics dominate your series count.

Illustrative estimate (assumed numbers, not a benchmark). Series = (buckets + 2) × label combinations.

• Assume 12 buckets, 20 routes, 5 status codes and 30 pods: (12 + 2) × 20 × 5 × 30 = 42,000 series.
• Trim to 6 buckets and remove pod: (6 + 2) × 20 × 5 = 800 series.

Use three levers:

• Trim buckets to the boundaries you alert on. histogram_quantile interpolates between bucket edges, so precision only matters near your SLO threshold. If the target is 300 ms, keep edges such as 0.1, 0.25, 0.3, 0.5, 1 and drop the 5 ms and 10 s tails that no query reads. You set this at instrumentation in the client library's bucket list.
• Strip high-cardinality labels from histograms only. Labels like pod or instance can stay on cheap counters and still be removed from histograms. You can do this at instrumentation. You can also use a metric_relabel_configs rule that matches __name__ against the histogram's name and sets pod to an empty replacement. An empty value removes the label. Only do this when another label still keeps the series distinct. Otherwise the series collide and you get ingestion errors.
• Reserve fine-grained histograms for SLO-bearing endpoints. For internal or batch paths, use a coarse histogram or a plain counter plus _sum.

Copyscrape_configs:
  - job_name: checkout-api
    static_configs:
      - targets: ['checkout-api:8080']
    metric_relabel_configs:
      # 1) Drop bucket edges no alert or dashboard reads (5 ms, 10 ms, 10 s tails).
      #    Buckets are cumulative, so histogram_quantile still works on the rest,
      #    as long as le="+Inf" is kept. Treat this as a stopgap: the real fix is
      #    a shorter bucket list in the client library.
      #    Some clients render 10 as "10.0", so match both forms.
      - source_labels: [__name__, le]
        regex: 'http_request_duration_seconds_bucket;(0\.005|0\.01|10(\.0)?)'
        action: drop

      # 2) Remove the pod label from this histogram only.
      #    replace sets pod to an empty string, and an empty label is removed.
      #    instance still differs per target, so series stay distinct.
      #    Without a distinguishing label they would collide and fail ingestion.
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_(bucket|sum|count)'
        target_label: pod
        replacement: ''
        action: replace

A scrape config that drops unused bucket boundaries from one histogram and removes the pod label from its _bucket, _sum and _count series, while other metrics from the same target keep the pod label.

What are the trade-offs between dropping, relabeling and aggregating labels?

Each layer removes cost at a different point in the pipeline, and each fails differently.

Dropping a label at scrape time is safe only when that label adds nothing to series identity, such as a constant version on a single-version fleet. If the label separates real values, use a recording rule with sum by (...) so the arithmetic is explicit.

Relabeling happens after the target has already exposed the series. The client library still holds every label combination in memory, the exporter still renders it, and the scraper still downloads it. Many teams find that a relabel rule they added as a stopgap stays in place for years.

Fixing the label at the source costs one pull request, and the savings compound. Dashboards and alerts query the series that actually exist, so they don't depend on hidden relabel logic. New engineers don't need to reconcile a raw metric with its aggregated twin. Incident debugging also skips the step of asking which layer changed the data before anyone looked at it.

What are the alternatives to trimming labels?

Trimming labels is not the only way to control cardinality cost. These alternatives solve parts of the same problem, often alongside trimming:

• Longer scrape intervals. On a sample-based bill, doubling the interval halves the samples from the affected jobs without losing any dimension. It does nothing for a series-based bill or for head-block memory, and it reduces time resolution for alerts and rate() windows.
• Shorter local retention. Reducing retention on the local server limits how many churned generations stay on disk and in long-range queries. Pair it with recording rules forwarded through remote_write if you still need long-term trends.
• Native histograms. Prometheus native histograms store a whole distribution in a single series instead of one series per bucket, which removes the bucket multiplier entirely. Support depends on your Prometheus server version and configuration, your client libraries and your remote backend, so confirm all three before you migrate. Queries also change, because native histograms are read differently from classic _bucket series.
• Summaries instead of histograms. A summary computes quantiles in the client and exposes a fixed, small set of series. The cost is that summary quantiles cannot be meaningfully aggregated across instances, so they suit per-instance views, not fleet-wide SLOs.
• Exemplars. Instead of putting a trace ID or request ID on a metric as a label, attach it as an exemplar. The series count stays bounded, and you can still jump from a latency spike to a representative trace. This requires exemplar support in the client library, the server and the dashboarding layer.
• Logs and traces for per-entity detail. User, tenant and request-level questions are often better answered by structured logs or tracing. As the trade-off table notes, this moves cost to another system rather than removing it.
• Scaling the backend instead. Sharding scrape targets across several Prometheus servers, or using a horizontally scalable long-term store, accepts the cardinality and pays for capacity. This is reasonable when the high-cardinality data is genuinely needed, but it adds operational complexity and does not lower a per-series bill.
• Separate retention or routing for expensive metrics. Some teams keep high-cardinality debug metrics on a short-retention local server only and never forward them. This is a targeted form of write_relabel_configs filtering rather than a separate technique, but it is worth planning explicitly.

In practice, label trimming at the source is the default because it lowers cost at every stage. The alternatives are better suited to cases where a high-cardinality dimension has real diagnostic value and you need to keep it somewhere.

How do you prevent cardinality regressions and save engineering time?

Guardrails should live in the same configuration. Prometheus scrape jobs accept sample_limit, plus label_limit, label_name_length_limit and label_value_length_limit. With these set, a deploy that suddenly emits a user ID as a label is rejected at scrape time. It never quietly becomes millions of billable series.

The failure mode is blunt. If a target exceeds sample_limit, Prometheus drops that entire scrape, not just the excess samples, and up goes to 0 for that target. A service that has started misbehaving can therefore go dark on your dashboards at the moment you most need its metrics. To handle this:

• Set limits with real headroom above normal output, for example comfortably above the peak of scrape_samples_post_metric_relabeling you've seen for that job.
• Alert on up == 0 and on increases in prometheus_target_scrapes_exceeded_sample_limit_total, so a tripped limit pages someone instead of looking like an outage.
• Alert on series growth per job. Compare sum by (job) (scrape_samples_post_metric_relabeling) against the same expression with offset 1d, and fire on a large ratio. This catches slow creep that never hits a hard limit.

Limits and alerts react after code ships. The cheapest fix comes earlier. Add a short checklist to code review for any new metric:

• What is the bounded set of values for each label?
• Is any label derived from user input, IDs, URLs or error strings?
• Does a histogram really need every label it carries?
• Who owns this metric?

A reviewer asking "what are the possible values of path?" spends a minute. A post-bill cleanup is far more expensive. Someone has to trace the spike to a metric, find the owning team, and ship and deploy a fix. Meanwhile, dashboards and alerts built on the bad label break as it disappears.

Copyscrape_configs:
  - job_name: checkout-api
    sample_limit: 20000             # peak post-relabel samples ~12k; leave headroom
    label_limit: 30                 # max labels per series
    label_name_length_limit: 128
    label_value_length_limit: 256   # long values often mean URLs or error strings
    static_configs:
      - targets: ['checkout-api:9090']
---
groups:
  - name: cardinality-guardrails
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 5m
        annotations:
          summary: '{{ $labels.job }}/{{ $labels.instance }} scrape failing (check sample_limit)'
      - alert: SampleLimitExceeded
        # Prometheus self-metric: any scrape rejected for exceeding sample_limit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        annotations:
          summary: 'A target exceeded sample_limit; its whole scrape was dropped'
      - alert: SeriesGrowthPerJob
        # Catch slow creep: per-job samples up more than 50% vs. one day ago
        expr: |
          sum by (job) (scrape_samples_post_metric_relabeling)
            / sum by (job) (scrape_samples_post_metric_relabeling offset 1d)
            > 1.5
        for: 30m
        annotations:
          summary: 'Series count for {{ $labels.job }} grew >50% in 24h'

A scrape job with sample and label limits set above its normal output, plus alert rules that flag failed scrapes, tripped sample limits and day-over-day series growth per job.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to prometheus high cardinality cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.