How does Datadog custom metrics billing count units?

Datadog does not bill custom metrics by metric name. The billable unit is a unique time series: one metric name combined with one distinct set of tag values. http.requests tagged endpoint:/checkout, status_code:200 is one series. The same metric with status_code:500 is a second series. Every new tag value creates another billable series, so a single metric name can account for thousands of units.

Consider an illustrative estimate. These inputs are assumed and were not measured. A request counter carries three tags:

  • endpoint with 50 values
  • status_code with 10 values
  • pod with 200 values
Tag set Theoretical maximum series
endpoint × status_code × pod 50 × 10 × 200 = 100,000
endpoint × status_code 50 × 10 = 500

Removing one tag reduces the ceiling by a factor of 200. This happens because tag cardinalities multiply. They do not add.

The 100,000 figure is an upper bound. Datadog counts only the combinations that actually report data. If most pods serve only a few endpoints, and most endpoints rarely return a 503, the real count will be far lower. The ceiling still matters because it is reachable. A traffic spike, a deploy that reschedules every pod, or a new endpoint can fill in combinations that were previously empty.

This is why the bill catches engineers off guard. Adding a tag looks like a small change in code review, but it multiplies the series count across every existing combination. Ephemeral infrastructure makes this worse: Kubernetes pods get new names on every rollout, so pod-name tags keep producing new series.

                  +---------------------------+
                  |  metric: http.requests    |
                  +-------------+-------------+
                                |
          +---------------------+---------------------+
          |                     |                     |
          v                     v                     v
 +-----------------+   +-----------------+   +-----------------+
 | tag: endpoint   |   | tag: status_code|   | tag: pod        |
 | 50 values       |   | 10 values       |   | 200 values      |
 +--------+--------+   +--------+--------+   +--------+--------+
          |                     |                     |
          +-------- x ----------+---------- x --------+
                                |
                                v
          +-------------------------------------------+
          | billable unit = 1 unique time series      |
          | (metric name + one combo of tag values)   |
          +---------------------+---------------------+
                                |
             +------------------+------------------+
             |                                     |
             v                                     v
 +-----------------------+             +-----------------------+
 | WITH pod tag          |             | WITHOUT pod tag       |
 | 50 x 10 x 200         |             | 50 x 10               |
 | = 100,000 max series  |             | = 500 max series      |
 +-----------------------+             +-----------------------+
  Actual count = only the combinations that report data.

Illustrative estimate with assumed inputs: tag cardinalities multiply, so dropping the 200-value pod tag cuts the theoretical maximum from 100,000 unique time series to 500.

Which tags drive cardinality, and how do you find them?

Each new pod name opens a fresh set of series, and the old ones stay on the bill for the period in which they reported. The same pattern shows up with a short list of tags:

Tag Why it is unbounded Usual fix
user_id, account_id Grows with your customer base Move to logs or traces; keep a plan or tier tag instead
request_id, trace_id One value per request Never tag metrics with these; correlate through APM
Raw URL path Path parameters such as /orders/8812 make each ID its own value Tag with the route template, e.g. /orders/{id}
pod_name, container_id Changes on every rollout or reschedule Aggregate by deployment, service, or namespace
Build SHA New value on every deploy Use a release version tag, or emit it once as an event

Audit before you change anything

Start with the usage pages in Plan & Usage. They show which metrics account for the most custom metric volume, so you work on the biggest offenders first. Then open each candidate in the Metrics Summary and look at the tag keys and how many distinct values each one has. This tells you which single tag is doing the multiplying.

Graph the datadog.estimated_usage.* metrics over several weeks. A step change that lines up with a deploy usually points to a newly added tag.

Before you drop a tag or a metric, find everything that queries it: dashboards, monitors, and SLOs. If a monitor groups by pod_name and you remove that tag, the monitor can stop evaluating without raising any error. Your bill goes down, and so does your alerting coverage. Rewrite those queries first, and only then cut the tag.

#!/usr/bin/env bash
# Find every monitor, dashboard and SLO that references a metric before you
# drop a tag from it. Read-only: this script only issues GET requests.
set -euo pipefail

METRIC="checkout.request.duration"   # metric you plan to change
TAG="pod_name"                       # tag you plan to drop
SITE="${DD_SITE:-datadoghq.com}"
API=""
AUTH=(-H "DD-API-KEY: ${DD_API_KEY}" -H "DD-APPLICATION-KEY: ${DD_APP_KEY}")

echo "== Monitors querying ${METRIC} (tag hit = also uses ${TAG}) =="
curl -sf "${AUTH[@]}" "${API}/monitor" |
  jq -r --arg m "$METRIC" --arg t "$TAG" '.[]
    | select(.query | contains($m))
    | "\(.id)\t\(if (.query | contains($t)) then "TAG HIT" else "-" end)\t\(.name)"'

echo "== SLOs querying ${METRIC} =="
curl -sf "${AUTH[@]}" "${API}/slo" |
  jq -r --arg m "$METRIC" '.data[]
    | select((.query // {} | tostring) | contains($m))
    | "\(.id)\t\(.name)"'

echo "== Dashboards querying ${METRIC} =="
# The list endpoint returns summaries only, so fetch each dashboard's full JSON.
for id in $(curl -sf "${AUTH[@]}" "${API}/dashboard" | jq -r '.dashboards[].id'); do
  body=$(curl -sf "${AUTH[@]}" "${API}/dashboard/${id}")
  if grep -q "$METRIC" <<<"$body"; then
    hit=$(grep -q "$TAG" <<<"$body" && echo "TAG HIT" || echo "-")
    echo -e "${id}\t${hit}\t$(jq -r '.title' <<<"$body")"
  fi
done
# Rewrite every TAG HIT query (e.g. group by service instead of pod_name)
# before removing the tag, or those monitors may silently stop evaluating.

A read-only audit script that uses the Datadog v1 API with curl and jq to list the monitors, SLOs and dashboards that reference a metric. It flags the ones that also use the tag you plan to drop, so you can rewrite those queries before cutting the tag.

How do histograms and distributions multiply series, and when are they worth it?

Metric type determines how many series one tag combination produces, so it matters as much as which tags you attach.

A DogStatsD histogram is aggregated by the Agent on each host. With default settings, every tag combination emits five series: avg, count, median, 95percentile and max. Each one is billed as its own custom metric. A distribution sends raw values to Datadog and aggregates them server-side across all hosts. By default it produces count, sum, min, max and avg. Turning on percentile aggregations adds p50, p75, p90, p95 and p99. Those percentiles are billed against the tag combinations you can query, which makes them the most expensive option per metric.

Type Series per tag combination What it can answer
Counter 1 Throughput, error counts
Gauge 1 Last value per interval (queue depth, memory)
Histogram 5 by default Per-host latency shape
Distribution 5, plus 5 with percentiles Globally accurate percentiles over any tag slice

It's tempting to switch latency metrics from distributions to histograms to save money. The problem is that percentiles cannot be combined. Averaging the p95 from 40 hosts does not give you the fleet's p95. A host handling a small amount of slow traffic carries the same weight as one serving most requests, so the result can look healthy while users see slow responses. An SLO built on that number can report compliance it isn't actually meeting.

Keep distributions with percentiles for metrics that back SLOs or user-facing latency alerts. Use counters or gauges for everything else. Before you remove percentiles from a distribution, cut the tags it carries, because each tag you drop reduces both the default series and the percentile series.

from datadog import initialize, statsd

initialize(statsd_host="localhost", statsd_port=8125)
tags = ["service:checkout", "env:prod"]

# Series cost per tag combination for each call, assuming default settings.
statsd.increment("checkout.requests", tags=tags)         # 1 series
statsd.gauge("checkout.queue_depth", 12, tags=tags)      # 1 series
statsd.histogram("checkout.latency_ms", 87.0, tags=tags) # 5: avg, count, median, 95percentile, max (per host)
statsd.distribution("checkout.latency_dist_ms", 87.0, tags=tags)  # 5 default, plus 5 if percentiles are enabled

# Why per-host p95s can't be averaged. Illustrative data, not measurements.
def p95(values):
    s = sorted(values)
    return s[int(0.95 * (len(s) - 1))]

# Nine quiet hosts serve fast traffic. One busy host serves most requests,
# and 10% of its requests are slow.
quiet_hosts = [[30.0] * 10 for _ in range(9)]
busy_host = [40.0] * 900 + [900.0] * 100
hosts = quiet_hosts + [busy_host]

# Every host gets the same weight here, whatever its traffic volume.
avg_of_p95 = sum(p95(h) for h in hosts) / len(hosts)

# This is what a distribution computes server-side from all raw values.
true_p95 = p95([v for h in hosts for v in h])

print(f"average of per-host p95: {avg_of_p95:.0f} ms")  # 117 ms, looks healthy
print(f"fleet-wide p95:          {true_p95:.0f} ms")    # 900 ms, what users actually see

The first part shows the series cost of each DogStatsD metric type for a single tag combination. The second part uses illustrative data to show that averaging per-host p95 values gives 117 ms when the fleet-wide p95 is 900 ms.

How do you fix metric cardinality at the source?

The cheapest series is the one that is never created. Cardinality is easiest to control in the code that emits metrics, before anything reaches the Agent.

Normalize routes into templates. Tag requests with the route pattern your framework matched, such as /users/:id/orders, not the raw path /users/48213/orders. Raw paths create a new series for every ID. Most web frameworks expose the matched template on the request object. Record that value, and fall back to a fixed value like unmatched for 404s so that scanners probing random paths cannot inflate your series count.

Put guardrails in a shared metrics wrapper. Route all instrumentation through one internal library. That library can drop tags known to be unbounded, such as user IDs, request IDs and full error messages. It can bucket continuous values, for example converting payload size into small, medium and large. It can also cap distinct values per tag and collapse any overflow into other. One fix in the wrapper protects every service that uses it.

Enforce a tag allowlist in code review. Keep a short, documented list of approved tag keys. Any new key should come with a stated value bound.

All of this costs less engineering time than repeated cleanup. Fixing cardinality after the fact means finding the series, tracing the code that emits it, and patching it across several repositories, often after a billing spike has already happened.

These controls also add friction for developers. If approving a new tag takes days, teams will either stop instrumenting or emit metrics directly and skip the wrapper. To avoid that, keep reviews fast, let teams add bounded tags without approval, and reserve strict review for keys whose values have no clear upper limit.

from datadog import statsd

# Reviewed allowlist of tag keys. Unlisted keys are dropped.
ALLOWED_TAGS = {"route", "method", "status_class", "payload_size", "region"}
MAX_VALUES_PER_TAG = 50  # per-process cap; overflow collapses into "other"
_seen = {}  # tag key -> values this process has already emitted

def _bucket_size(num_bytes):
    # Turn a continuous value into three bounded buckets.
    if num_bytes < 10_000:
        return "small"
    if num_bytes < 1_000_000:
        return "medium"
    return "large"

def _safe_tags(tags):
    out = []
    for key, value in tags.items():
        if key not in ALLOWED_TAGS:
            continue  # drops user_id, request_id, error_message, etc.
        if key == "payload_size":
            value = _bucket_size(value)
        seen = _seen.setdefault(key, set())
        if value not in seen and len(seen) >= MAX_VALUES_PER_TAG:
            value = "other"  # stops a runaway tag before it reaches the Agent
        else:
            seen.add(value)
        out.append(f"{key}:{value}")
    return out

def count_request(route_template, method, status, payload_bytes):
    # route_template is the framework's matched pattern, e.g. "/users/:id/orders".
    # A missing template (404) becomes "unmatched", so scanner paths add no series.
    statsd.increment("web.request.count", tags=_safe_tags({
        "route": route_template or "unmatched",
        "method": method,
        "status_class": f"{status // 100}xx",
        "payload_size": payload_bytes,
    }))

This shared DogStatsD wrapper uses route templates, applies a tag-key allowlist, buckets payload size, and collapses overflow tag values into "other" before any metric is sent.

When should you use Metrics without Limits to control indexed tags?

Metrics without Limits separates two things that are easy to conflate. Ingested volume is every series the Agent or API sends. Indexed volume is the set of tag combinations you can actually query. By configuring an allowlist of tags for a metric, you keep sending the full payload but only the tags you choose stay queryable. Datadog aggregates the rest away at query time.

Do not assume un-indexed series are free. Ingested and indexed custom metrics can be priced separately, and the rates and allotments depend on your contract. Check how each is billed under your plan before you count savings. Then compare the usage pages before and after a change, rather than trusting a projection.

Tag configuration (Metrics without Limits) Fixing at the source
Speed Minutes, in the UI or API Code change, review, deploy
Requires code ownership No Yes
Ingested volume Unchanged Reduced
Reversibility Re-add tags in config Redeploy
Risk Lost query dimensions Lost dimensions, plus emitter churn

It works best as a fast fix for metrics you cannot easily change. Examples are integration metrics, third-party libraries, or a team that will not get to a refactor this quarter. Use it to stop the bleeding, then file the source-level fix.

Avoid it for tags you reach for during incidents. If you drop pod or customer from the index, you lose the ability to slice by that tag in dashboards and investigations, including when you look back at past incidents. Before you rely on re-adding a tag to recover history, confirm how your account handles it. Treat any removal of an incident-critical dimension as a decision for the on-call owners, not a cleanup task.

How are billable hosts counted, and how can you reduce them?

Host billing is not usually an average. On many plans, Datadog samples the number of distinct hosts each hour and bills on a high-percentile or high-water-mark view of those hourly counts. Your contract defines the exact method, so check it before you model anything. The consequence is the same either way: a few hours of peak load set the price for the month. Autoscaling bursts, CI runner pools, and nightly batch fleets can count for more than the steady baseline that runs all month.

Reduction levers

  • Skip the Agent on low-value ephemeral machines. A CI runner that lives for ten minutes still counts in the hour it ran. If nobody looks at its host metrics, it adds cost and no visibility.
  • Consolidate onto fewer, larger nodes where your workloads tolerate it. Ten small nodes become ten billable hosts at peak; three large ones become three.
  • Count containers and integrations separately from Agent hosts. Containers are typically metered with their own allotment or overage. Cloud integrations can report hosts that have no Agent installed. Reducing Agent hosts does nothing about either.
  • Apply resource filters on cloud integrations. Restrict collection by tag so the integration does not pull in instances nobody monitors.

Trade-offs

Larger nodes widen the blast radius when one fails. They also make bin-packing harder and weaken noisy-neighbor isolation. Dropping the Agent from CI and ephemeral hosts creates blind spots in exactly the places where flaky failures, resource exhaustion, and disk-full errors tend to show up first. Before you cut coverage there, decide how you will debug those failures without it.

What are the alternatives to cutting metrics, and how do you keep spend owned?

those failures without them. The same tradeoff applies to every option that cuts cost without changing what gets emitted.

Alternative What it changes Main risk
Contract negotiation Rate per unit, committed volume Usage growth eventually overtakes the discount
Blind sampling Fraction of data points or hosts reported Rare, high-signal events disappear first
Turning monitoring off Coverage Incidents in unmonitored areas run longer
Moving data to logs or APM Which product bills for the data Spend moves to another line item instead of shrinking

Moving high-cardinality data such as per-user or per-request dimensions into logs or traces is often the right modeling choice, because those signals are naturally per-event. It does not cut spend by default, though. Ingestion, indexing, and retention have their own pricing. Compare total observability spend before and after the move, not the custom metrics line alone.

Keep spend owned

  • Attribute usage. Enforce team and service tags so usage attribution reports map cost to an owner. Unowned usage tends to grow.
  • Monitor estimated usage. Put monitors on the estimated custom metric usage metrics, grouped by owner where possible. A cardinality regression from a new tag then shows up within days, not on the invoice.
  • Review new metrics and tags. A short checklist in code review covers the questions that matter: which tags are attached, what their maximum distinct values are, and whether a histogram or distribution is needed.

Allotments, committed versus on-demand rates, and counting rules differ between contracts and change over time. Check any savings estimate against your own usage data before you count the savings.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce datadog costs, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.