What actually drives a Datadog bill, and where should a cost review start?
Most Datadog bills grow through three mechanisms, and each one scales with engineering decisions rather than with how many dashboards you have:
| Driver | What multiplies cost | Typical cause |
|---|---|---|
| Custom metric cardinality | Unique combinations of metric name and tag values | Tags such as user_id, pod_name or full URL paths |
| Logs indexed by default | Events stored in searchable indexes, in addition to ingestion | Catch-all index with no exclusion filters |
| Unplanned hosts and containers | Billable hosts and containers above any allotment | Agents on CI runners, autoscaling peaks, ephemeral nodes |
Audit these in order of dollar impact. Removing an unused dashboard saves nothing, because dashboards are not what gets billed. Negotiating before you fix cardinality locks in a commitment sized to waste.
Illustrative estimate (assumed numbers, not a benchmark). Assume one latency metric is tagged with 50 endpoints, 5 status codes and 40 pods.
- Formula: series ≈ endpoints × statuses × pods = 50 × 5 × 40 = 10,000
- Cost ≈ (series − allotment) × per-series rate
- With an assumed allotment of 0 and an assumed rate of $0.05, the cost is about $500 per month.
- Dropping the pod tag reduces it to 250 series.
This checklist is for backend and platform engineers who own instrumentation and Agent configuration. Run it in three situations:
- After unexplained bill growth.
- When a new environment comes online.
- After a migration, such as moving to Kubernetes or changing logging libraries.
Billing models, unit definitions and allotments vary by plan and contract, and they change over time. Check your own contract and the usage pages in your account before acting. Treat every figure here as illustrative.
How does custom metrics pricing count series?
Datadog bills custom metrics by distinct time series, not by metric name. A series is one unique combination of metric name and tag values. One metric emitted with many tag combinations can cost far more than dozens of metrics with few tags. Cardinality, the number of distinct values each tag takes, is what drives the bill.
Illustrative estimate
Assume a counter http.requests tagged with 10 endpoints and 5 status codes, where every combination actually occurs.
Tags on http.requests |
Calculation | Distinct series |
|---|---|---|
| endpoint, status | 10 × 5 | 50 |
| + pod_name (40 pods) | 50 × 40 | 2,000 |
| + user_id (10,000 active users) | 2,000 × 10,000 | up to 20,000,000 |
Only combinations that are actually reported count, so the last row is an upper bound. Even at a small fraction of it, a single unbounded tag dwarfs everything else. Pod names add a second problem: every rolling deploy creates new names. The series count within a billing period therefore grows with deploy frequency, not only with replica count.
Finding the worst offenders
Start on the usage page. It shows which product lines and time periods account for custom metric volume. Then open the metrics summary and sort by series count to find the handful of metrics that dominate. For each one, check the tag keys and how many distinct values each key has. A tag with thousands of values, such as an ID or a pod name, is the usual culprit.
Then decide tag by tag. Is this tag used in a dashboard, monitor or regular query? Usually a few tags carry the real analytical value and the rest are attached by default. Record the decision for each metric so the cleanup in the next section is deliberate rather than guesswork.
How can you cut custom metric cardinality in code and config?
Start with the tags that have no upper bound: user IDs, request IDs, full URLs with path parameters, session tokens, pod names on short-lived workloads. Each distinct value creates a new series. A single latency metric tagged with customer_id scales with your customer count, multiplied by every other tag combination on that metric.
- Drop or bucket at the source. Replace
customer_id:8f3a…withcustomer_tier:enterprise, and replace raw paths like/orders/91823with the route template/orders/{id}. A tier or template tag usually has a handful of values. An ID tag grows with the business. Do this in the instrumentation library or a shared wrapper so one engineer's new tag cannot slip past review. - Move per-entity detail out of metrics. If you need to answer "what happened to this customer's request," that question belongs to traces or structured logs, which carry the ID on each event. Keep metrics for aggregates such as rates, percentiles and saturation.
- Use tag configuration controls. Datadog lets you specify which tags stay queryable for a given metric and exclude the rest from indexing. This helps when you cannot change the emitting code quickly. Which controls you have and how they are billed depends on your plan and contract, so check your billing terms before relying on them.
These changes have costs. Dropped tags cannot be restored for data already collected, so confirm that no dashboard or monitor depends on a tag before you remove it. Bucketing makes per-customer debugging less precise. Moving detail into logs or traces shifts spend to those products rather than removing it, so estimate the new volume before you migrate.
from datadog import statsd # DogStatsD client from the datadogpy package
# Bounded lookup: a handful of tiers instead of one value per customer.
TIER_BY_PLAN = {"free": "free", "team": "standard", "business": "standard", "ent": "enterprise"}
def record_latency_bad(customer_id: str, raw_path: str, ms: float) -> None:
# High cardinality: every customer ID and every concrete path
# (e.g. /orders/91823) creates a new series, multiplied by all other tags.
statsd.histogram(
"checkout.request.latency",
ms,
tags=[f"customer_id:{customer_id}", f"path:{raw_path}"],
)
def record_latency_good(plan: str, route_template: str, ms: float) -> None:
# Bounded tags: tier has a few values, and the route template
# (e.g. /orders/{id}) comes from the router, not the request URL.
tier = TIER_BY_PLAN.get(plan, "other") # unknown plans collapse into one bucket
statsd.histogram(
"checkout.request.latency",
ms,
tags=[f"customer_tier:{tier}", f"route:{route_template}"],
)
# Per-customer detail (customer_id, request_id) belongs on the trace span
# or a structured log line for this request, not on the metric.
# Example call from a request handler:
# record_latency_good(plan="ent", route_template="/orders/{id}", ms=182.4)
Contrasts a latency metric tagged with unbounded customer IDs and raw paths against one tagged with a bucketed customer tier and a route template, which keeps the series count bounded.
How does log indexing differ from log ingestion in Datadog billing?
Datadog bills log ingestion and log indexing as separate line items. Ingestion is charged by volume, and indexing is charged per event, at a rate that depends on how long the index retains it. Indexing usually costs more, and it is the part you control after the data arrives. So the main lever is to ingest broadly and index selectively.
The pipeline runs in this order:
- Agent or forwarder ships logs, and they are billed as ingested.
- Processing pipelines parse the logs and attach attributes.
- Metrics generated from logs read every ingested log, including logs that are later excluded.
- Index exclusion filters drop or sample logs by query before they are stored.
- Indexes store the remaining logs, which are searchable and billed per event.
- Archives send ingested logs, including excluded ones, to your own object storage for later rehydration.
Exclusion filters work best on sources that are predictable, high-volume and low-value. Examples are load balancer health checks, debug output left on in production, and successful 2xx access logs. You can exclude these entirely or sample them at a small percentage so you keep representative examples.
Before you exclude a source, create log-based metrics for what your dashboards and monitors depend on, such as request counts by status, error rates and latency percentiles. Generation happens upstream of exclusion, so those monitors keep firing after the raw events stop being indexed. Each generated metric is billed as a custom metric, so keep its tags to the low-cardinality set from the previous section.
The trade-off is visibility. Sampling can drop the one odd request you need during an incident. Exclude only patterns you can describe precisely, leave unexpected or unparsed logs indexed, and keep archives so you can rehydrate excluded data when an investigation needs it.
# Step 1: create a log-based metric BEFORE excluding anything.
# It counts every ingested nginx request by status, including requests the index later drops.
curl -X POST "" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d '{
"data": {
"type": "logs_metrics",
"id": "nginx.requests.count",
"attributes": {
"compute": { "aggregation_type": "count" },
"filter": { "query": "source:nginx" },
"group_by": [
{ "path": "@http.status_code", "tag_name": "status_code" },
{ "path": "service", "tag_name": "service" }
]
}
}
}'
# Step 2: add exclusion filters to the "main" index.
# sample_rate is the fraction EXCLUDED: 1.0 drops all matches, 0.95 keeps about 5%.
# Warning: PUT replaces the index's whole exclusion_filters list, so include any existing filters.
curl -X PUT "" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
-d '{
"filter": { "query": "*" },
"exclusion_filters": [
{ "name": "drop-health-checks", "is_enabled": true,
"filter": { "query": "source:nginx @http.url_details.path:\"/healthz\"", "sample_rate": 1.0 } },
{ "name": "sample-successful-access", "is_enabled": true,
"filter": { "query": "source:nginx @http.status_code:[200 TO 299]", "sample_rate": 0.95 } }
]
}'
This example first creates a log-based count metric for nginx requests by status, then adds index exclusion filters that drop health checks and keep about 5% of successful access logs, so error-rate monitors still see all traffic.
How should you design log indexes, retention and daily quotas?
Start by routing logs into a small number of indexes based on how you use them, not by team or service. A typical split is a short-retention index for high-volume application and access logs, a longer-retention index for audit and security events, and a catch-all for anything unmatched. Indexing is billed per event and the price rises with the retention period, so a 30-day window for audit logs makes sense, while debug chatter rarely needs more than a few days.
Set a daily quota on each index to cap runaway volume, such as a deploy that logs inside a hot loop. Quotas have a known failure mode: the spikes that hit the limit usually happen during incidents, and once an index reaches its quota it stops indexing for the rest of the day. Alert well before the limit, not when you reach it. Decide in advance whether on-call engineers may raise the quota, and route critical sources to an index with its own quota so a noisy neighbour cannot exhaust it.
Archive everything to object storage, including excluded logs. Archived logs are not searchable until you rehydrate them, which means a scan job with its own cost and a wait.
| Log state | Search speed | Cost profile | Incident usefulness |
|---|---|---|---|
| Indexed | Immediate | Highest, priced per event and retention | Primary tool for live debugging |
| Excluded but archived | Not searchable | Ingestion plus cheap storage | None until rehydrated |
| Rehydrated | Slow to become available | Scan cost plus temporary indexing | Postmortems and audits, not live triage |
Why are you paying for hosts and containers you never chose to monitor?
Infrastructure billing counts every host and container that reports through the agent during the billing period, not the fleet you intended to monitor. Host counts are typically taken hourly and summarized as a high percentile across the month. A burst that runs for a few hours can therefore move the bill much less than a steady baseline of hosts nobody remembers installing an agent on. That baseline is where the waste usually sits.
Checklist: find hosts you did not choose
- CI runners. Runner images built from a base AMI or VM template that includes the agent register as hosts, and each short-lived runner can appear as a distinct host. Check host lists for naming patterns from your CI system.
- Autoscaling groups with no floor discipline. Groups that scale up quickly but scale in slowly hold hosts longer than the workload needs.
- Duplicate agents. A host-level agent plus a DaemonSet agent on the same Kubernetes node, or a sidecar agent inside pods, can report the same machine twice or inflate container counts.
- Dev and sandbox environments. Personal clusters and forgotten test stacks often copy production manifests verbatim, agent included.
Control installation in deployment configuration
Make the agent an explicit opt-in instead of something every image inherits. Remove it from shared base images. Scope DaemonSets with node selectors. Use the agent's container include/exclude filters so infrastructure pods and batch jobs are not tracked by accident. Enable integrations per environment rather than globally.
The trade-off
Removing agents from CI and staging saves money but leaves you without data at the moment a flaky test or failed deploy needs diagnosing. A middle ground is a minimal agent profile in those environments: basic host metrics with no APM or process collection. Another option is enabling full collection only on the pipelines you are actively debugging.
# Fragment of an agent DaemonSet for a staging cluster (not a complete manifest).
# Goal: a deliberate, minimal agent profile instead of a copy of production.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: datadog-agent
namespace: monitoring
spec:
template:
spec:
# Run only on nodes explicitly labelled for monitoring, not every node.
# CI runner and sandbox node pools stay unlabelled and are never counted.
nodeSelector:
monitoring/agent: "enabled"
containers:
- name: agent
image: gcr.io/datadoghq/agent:7
env:
- name: DD_API_KEY
valueFrom:
secretKeyRef:
name: datadog-secret # placeholder secret name
key: api-key
# Exclude everything by default, then include only app namespaces,
# so system pods and batch jobs are not tracked by accident.
- name: DD_CONTAINER_EXCLUDE
value: "name:.*"
- name: DD_CONTAINER_INCLUDE
value: "kube_namespace:checkout kube_namespace:payments"
# Minimal profile: host metrics and checks stay on, so failed
# deploys are still diagnosable. Heavier features are off here.
- name: DD_LOGS_ENABLED
value: "false"
- name: DD_APM_ENABLED
value: "false"
- name: DD_PROCESS_AGENT_ENABLED
value: "false"
A partial staging DaemonSet that limits the agent to labelled nodes and specific namespaces, keeping basic host metrics while turning off logs, APM and process collection.
What are the alternatives to cutting telemetry, and when should you avoid aggressive cuts?
Cutting data is only one option, and each option fails differently.
| Option | Cost effect | Reversibility | Debugging impact | Engineering effort |
|---|---|---|---|---|
| Reduce at the source (drop tags, sample, lower log levels) | High; data is never sent | Low; requires a deploy, and past data is gone | Highest risk if a dimension is needed later | Medium; code changes across services |
| Datadog-side tag and index controls | Medium to high on indexing and custom metric counts | High; a config change | Moderate; raw data may still exist upstream | Low |
| Move detail to traces or logs | Medium; replaces high-cardinality metric tags | Medium | Often better; per-request context is kept | Medium |
| Archive only, rehydrate on demand | High on indexing | High for future data; past queries need rehydration | Slow access during incidents | Low to medium |
| Contract negotiation after the audit | Variable; can lower unit prices | Tied to the contract term | None | Low; mostly procurement time |
Prefer the reversible options while you learn which data matters. Source-side cuts should come last, once usage data shows a tag or log stream is never queried.
When to hold back
- During incident-prone periods, such as migrations, launches or a quarter with repeated outages. That is when you lose visibility at the worst moment.
- For dimensions named in on-call runbooks or alert queries. Removing
customer_tierfrom a metric can silently break the alert built on it. - When the savings are small. Illustrative estimate: saving $300/month is wiped out if one incident takes four engineers an extra three hours because the right data is missing.
Under-instrumentation usually costs more than the bill it saves. The cost shows up as longer outages, not as a line on the invoice.
How do you keep Datadog costs from creeping back up?
Cost drifts back up through many small additions: a new tag on a metric, a debug log level left on after a release, a sidecar that ships its own logs. Treat spend as a property that is checked on every change.
Make every unit of spend attributable
Put team, service and env tags on all metrics, logs and hosts, and enforce them in shared libraries and agent config so individual services cannot skip them. With these tags, Datadog's usage attribution views can break spend down by owner. A team that can see its own line item usually fixes it without being asked, and nobody has to work out who owns an unlabeled metric.
Review on a schedule
Put a short monthly usage review on the calendar. Compare custom metric counts, indexed log volume and host counts against the previous month, broken down by team. Look for jumps. Gradual growth is normal.
Gate new cardinality at review time
Add a CI check that scans metric emission code and log pipeline config for tag keys on a deny list, such as user_id, request_id, session_id or raw URL paths. It should fail with a message explaining why. The code example accompanying this section shows that pattern.
At review time, removing a bad tag is a one-line change. After it ships, the same tag means a cleanup across dashboards, monitors and other teams' queries, usually started by an unexpected bill.
The trade-off is that strict gates add friction. They can also push engineers to drop instrumentation instead of arguing with a linter. Keep the deny list short. Allow a reviewed exception with an inline justification and an owner, and audit those exceptions during the monthly review.
#!/usr/bin/env python3
"""Fail CI when a change adds a high-cardinality tag key to metrics or log config.
Usage (changed files only):
git diff --name-only origin/main...HEAD | xargs python3 -I check_tags.py
"""
import re
import sys
# Keep this short: only keys whose values are effectively unbounded.
DENY = ("user_id", "request_id", "session_id", "trace_id", "url_path")
# Matches a deny-listed key used as a tag: "user_id:42", f"request_id:{rid}",
# or an unquoted YAML list item such as `- session_id:abc`.
TAG_RE = re.compile(r"(?:[\"']|^\s*-\s*)\s*(" + "|".join(DENY) + r")\s*:")
# A reviewed exception must name an owner and a reason on the same line.
EXCEPTION_RE = re.compile(r"cardinality-ok:\s*owner=\S+\s+reason=\S+")
def main(paths):
failures = 0
for path in paths:
with open(path, encoding="utf-8", errors="replace") as fh:
for lineno, line in enumerate(fh, 1):
match = TAG_RE.search(line)
if not match or EXCEPTION_RE.search(line):
continue
failures += 1
print(
f"{path}:{lineno}: tag key '{match.group(1)}' is high-cardinality. "
"On a metric, every distinct value is a new billable time series. "
"Put it in the log message or a trace instead, or after review add "
"'# cardinality-ok: owner=<team> reason=<why>' to this line."
)
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
A minimal CI check that fails a pull request when changed files tag metrics or logs with deny-listed, unbounded keys, unless the line has an inline exception that names an owner and a reason.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to reduce datadog costs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.