What actually drives a Datadog bill for custom metrics and logs?
Two mechanisms account for most of the custom metrics and log spend that engineers can change directly.
Custom metrics are billed per time series, not per metric name. Each unique combination of metric name and tag values counts as a separate series. Tags multiply rather than add.
Illustrative estimate, with assumed inputs: one latency metric tagged by 20 endpoints, 5 status codes and 300 pods gives 1 × 20 × 5 × 300 = 30,000 series. Pod name is already the largest factor because pods churn on every deploy and scale event, and each new pod name creates new series. Now add a user_id tag with an assumed 10,000 active users. The worst case is 300 million series. Real counts are lower because not every user hits every endpoint on every pod, but the growth is unbounded. request_id and full URL paths with embedded IDs behave the same way.
Metric type matters too. Distributions and histograms emit several aggregations or percentiles per tag combination, such as count, average, max and configured percentiles. The same tags therefore cost several times more on a histogram than on a gauge or count.
Logs are billed in two separate stages. Ingestion and indexing are priced separately. A log you ingest but never index still costs money, and indexing cost rises with the retention period you choose.
| Lever | What drives cost | What you control |
|---|---|---|
| Custom metrics | Unique tag combinations × aggregations per type | Tag cardinality, metric type |
| Log ingestion | Volume sent | What the Agent forwards |
| Log indexing | Events indexed × retention | Index filters, retention period |
Pricing, SKUs and committed-use terms vary by contract. This article does not quote prices.
Who should own observability spend, and when is cleanup worth the effort?
The people who can lower this bill are the ones who control what gets emitted and where it goes: backend engineers who own instrumentation libraries and metric naming, and platform engineers who own Agent configuration, log pipelines, and index settings. Finance and procurement can negotiate rates. They cannot change how many tag combinations a service produces or how many debug lines reach an index.
That is why emission and routing changes usually beat contract negotiation. A better rate lowers the price of each unit, while cardinality and volume keep growing with every new service and tag. Removing a high-cardinality tag or excluding noisy logs reduces the number of units billed, and that reduction applies to all future growth as well.
When to start
- Usage is rising faster than traffic or headcount. This often points to cardinality or verbosity creep.
- New services are onboarding. Defaults set now are cheaper than cleanup later.
- A sudden spike follows a deploy. A new tag such as a request ID or user ID can multiply time series overnight.
Is it worth the time?
Compare savings against the engineering cost:
net monthly value = (monthly spend on the offender × expected reduction) − (engineer hours × loaded hourly cost ÷ months to amortize)
Illustrative estimate (assumed numbers, not a benchmark): One offending metric costs $3,000/month and a fix removes 70% of it, saving $2,100/month. The fix takes 16 hours at $120/hour, a one-time cost of $1,920, so it pays back in about one month. An offender costing $100/month with the same fix would take more than a year to pay back.
Spend tends to concentrate in a few sources, so rank by cost and fix the top handful before auditing anything else.
How do you find the metrics and logs costing you the most?
Building that ranked list starts in the usage views rather than in your code.
Rank custom metrics by series count
Datadog bills custom metrics by unique time series, not by metric name. In the Metrics Summary view, sort by volume (distinct series count) so the names with the most tag combinations rise to the top. A single metric tagged with user_id, request_id, or container ID can produce more series than hundreds of well-behaved metrics. Open each top entry and look at which tag keys drive the cardinality. Where the tag configuration view shows which tags were actually queried, compare that against the full tag set. Tags that are emitted on every point but never filtered or grouped on are the cheapest wins.
Usage Attribution shows the same spend broken down by team, service, or environment tag. That tells you who to ask before you change anything.
Find the loudest log sources
For logs, group ingested volume by service, source, and status, and check which indexes retain the most events. The estimated usage metrics make this easy to chart. Log Patterns often show that one repetitive message, such as a health-check line or a per-request debug log, accounts for a large share of volume.
Cross-check before cutting
Before you drop a tag or exclude a log, search dashboards, monitors, SLOs, and log-based metrics for references to it. A tag that no one queries interactively may still feed a paging monitor. Write down what each candidate fix would save and what depends on it. That short annotated list is what the next sections work from.
How do you reduce custom metrics cost without losing visibility?
The cheapest series is the one never emitted. Custom metric volume grows with the number of unique tag-value combinations, so the fix is to bound every tag's possible values before the data leaves your process.
Start with request paths. /users/8812/orders/77 and /users/9043/orders/12 should both report as /users/{id}/orders/{id}. Use your framework's matched route template rather than a regex over raw URLs, because the router already knows the shape. Apply the same idea to other unbounded values. Replace customer IDs with a plan tier, error messages with an error class, and raw status codes with 2xx/4xx/5xx where the exact code is never queried. Drop host, pod_name, or container_id from metrics you only read at the service level. Those tags multiply series with every deploy and autoscaling event.
The wrapper shown below enforces this in code. It rejects any tag key outside an allowlist and normalizes paths, so a new engineer cannot ship user_id as a tag by accident.
Treat distributions as a deliberate choice. Each percentile you enable on a distribution metric adds series for every tag combination. Enable p95 and p99 only where an SLO or a dashboard actually reads them.
For high-volume metrics you cannot change at the source quickly, use Metrics without Limits to configure which tags stay queryable. Datadog still ingests the full data but indexes only the tags you list. Ingested and indexed custom metrics are billed separately, so check your plan's pricing rather than assuming the dropped combinations become free.
import re
from datadog import statsd # DogStatsD client from the `datadog` package
# Every key here must have a small, known set of values.
ALLOWED_TAG_KEYS = {"service", "env", "route", "method", "status_class", "plan_tier", "error_class"}
# Fallback only: collapse numeric and UUID path segments to {id}.
_ID_SEGMENT = re.compile(r"/(\d+|[0-9a-fA-F-]{36})(?=/|$)")
def normalize_path(raw_path, route_template=None):
# Prefer the router's matched template; the router already knows the shape.
return route_template or _ID_SEGMENT.sub("/{id}", raw_path)
def _build_tags(tags):
unknown = set(tags) - ALLOWED_TAG_KEYS
if unknown:
# Fail in tests/CI instead of quietly shipping user_id or similar.
raise ValueError(f"tag keys not in allowlist: {sorted(unknown)}")
return [f"{k}:{v}" for k, v in tags.items()]
def record_request(method, raw_path, code, duration_ms, route_template=None, **tags):
tags.update(
route=normalize_path(raw_path, route_template),
method=method.upper(),
status_class=f"{code // 100}xx", # 2xx/4xx/5xx, not every exact code
)
dd_tags = _build_tags(tags)
# No host/pod/container tags added: these metrics are read per service.
statsd.increment("app.http.requests", tags=dd_tags)
statsd.distribution("app.http.request.duration_ms", duration_ms, tags=dd_tags)
# Example: /users/8812/orders/77 is reported as route:/users/{id}/orders/{id}
record_request("get", "/users/8812/orders/77", 200, 42.0,
service="checkout", env="prod", plan_tier="pro")
A Python DogStatsD wrapper that rejects tag keys outside an allowlist, prefers route templates over raw paths, and buckets status codes so every tag has a bounded set of values.
How does log ingestion filtering work from the Agent to the index?
A log line passes through four stages, and each one controls a different part of the bill:
App ──▶ Agent ──▶ Ingestion (pipelines, log-based metrics) ──┬──▶ Indexes (exclusion filters, retention)
└──▶ Archive (object storage, rehydration)
Application. Volume you never emit costs nothing. Set production log levels to INFO or WARN and make DEBUG an opt-in per service or request. Raising the level after an incident is cheaper than filtering DEBUG later in the pipeline.
Agent. Processing rules run on the host before anything ships, so this is the last point where dropping a line saves ingestion cost. Use exclude_at_match for health checks and readiness probes. Use mask_sequences to redact tokens or emails before they leave your network. The example below does both.
Ingestion. Every log that reaches Datadog is billed for ingestion, including logs that are never indexed. Use that volume before you discard it. Log-based metrics are computed from all ingested logs, regardless of index filters, so you can count 5xx responses or measure latency from a log and then exclude the raw lines.
Indexes. Exclusion filters reduce indexing cost, not ingestion cost. Sample by level and outcome. For example, keep every error, keep 10% of successful requests, and drop routine access logs. Put noisy sources in a separate index with short retention.
Archives. Route ingested logs to your own object storage to meet compliance requirements. Rehydrate a time window only when an investigation needs it. Rehydrated logs are indexed again and billed again, so keep the window narrow.
# /etc/datadog-agent/conf.d/checkout.d/conf.yaml
# Requires logs_enabled: true in datadog.yaml.
# Rules run on the host, in order, before logs ship.
# Patterns use Go regex syntax (RE2), so lookarounds are not supported.
logs:
- type: file
path: /var/log/checkout/app.log
service: checkout
source: python
log_processing_rules:
# Drop load balancer and Kubernetes probe traffic. Excluded lines
# never leave the host, so they add no ingestion cost.
- type: exclude_at_match
name: exclude_health_checks
pattern: '(GET|HEAD) /(healthz|readyz|livez)'
# Redact bearer tokens in the authorization field before they
# leave your network. Only the matched sequence is replaced.
- type: mask_sequences
name: mask_auth_tokens
replace_placeholder: 'authorization=Bearer [MASKED]'
pattern: 'authorization=Bearer [A-Za-z0-9._~+/=-]+'
# Redact email addresses wherever they appear in the line.
- type: mask_sequences
name: mask_emails
replace_placeholder: '[MASKED_EMAIL]'
pattern: '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
This Agent log configuration drops health-check and readiness-probe lines on the host and masks bearer tokens and email addresses before logs are shipped and billed for ingestion.
When should you avoid dropping tags, sampling logs, or relying on archives?
Rehydration works for audits and compliance requests, where waiting minutes or hours is acceptable. It is a poor fit for incident response. If your plan for debugging a live outage is "rehydrate the archive," you are paying twice and waiting while the incident continues.
Dropping tags has a similar limit. A dimension you never stored cannot be added to past data. If you remove customer_id or pod_name to cut custom metrics cost, a later question like "which tenant saw the latency spike last Tuesday?" has no answer. Before you remove a tag, check whether anyone filtered or grouped by it during recent incidents.
Aggregation can also hide the signal. A p99 latency rolled up across all pods can look healthy while one node on a bad host serves slow responses to a fraction of traffic. Keep at least one dimension that maps to your failure domains: node, zone, or deployment version.
Uniform sampling of log ingestion has the same weakness. A 10% sample keeps routine noise and drops nine in ten of the rare errors you need. Keep errors and failed requests at 100%, and sample success-path and debug logs by level and outcome.
Logs-to-metrics rules and tag configurations live in the vendor UI, outside your codebase. They can drift without review, and a renamed attribute can break a dashboard or monitor without any warning.
The safe target is a minimal set of dimensions that matches how your team actually investigates failures. Base it on recent postmortems rather than guesses.
What are the alternatives to cutting telemetry, and how do you keep savings from regressing?
Before cutting more telemetry, compare the levers side by side. They differ mainly in what you give up and how easily you can get it back.
| Cost lever | Effort | Visibility lost | Reversibility |
|---|---|---|---|
| Contract or commitment negotiation | Low engineering, high coordination | None | Fixed for the contract term |
| Tag configuration in Datadog | Low | Dropped tags can't be queried | High; data still arrives |
| Source-level cardinality reduction | Medium to high | Dropped dimensions gone at source | Low; needs a code change and redeploy |
| Log index exclusion and sampling | Low | Excluded events aren't searchable | Medium; affects only future logs |
| Logs-to-metrics | Medium | Raw events, if you stop indexing them | Medium |
| Archive-only retention | Low | Live search; rehydration adds delay and cost | High for archived data |
| Pre-ingestion filtering (Agent) | Medium | Everything filtered | None for dropped data |
Keeping savings from regressing
Cleanups rarely fail because the original fix was wrong. They fail because new code brings the same problem back. Three controls prevent that:
- Attribute usage per team. Require
teamandservicetags so each team can see its own share of spend. - Alert on usage, not invoices. Put monitors on Datadog's estimated usage metrics so a cardinality spike shows up within hours rather than at month end.
- Check new metric tags in code review. A CI step compares tag keys in changed metric calls against an allowlist. It flags keys that are usually unbounded, such as
user_id,request_id,email, raw URL paths, or anything holding interpolated strings. To pass, the author either maps the value to a bounded set or justifies the exception in the review.
The check costs a reviewer a few seconds. Without it, the cost is an incident-style investigation weeks later: finding the offending metric, tracing it to a deploy, and changing dashboards that came to depend on the tag. Catching the tag at review time stops that cleanup from repeating every quarter.
#!/usr/bin/env python3
# CI check: flag risky tag keys in metric calls added by this branch.
# Usage: python check_metric_tags.py origin/main
import re, subprocess, sys
# Keys whose values come from a small, known set. Extend via review.
ALLOWED = {"team", "service", "env", "region", "endpoint", "status_class"}
# Keys that almost always carry one value per user, request or URL.
UNBOUNDED = {"user_id", "request_id", "session_id", "email", "url", "path"}
METRIC_CALL = re.compile(r"\.(increment|decrement|gauge|histogram|distribution|timing)\(")
# Matches tag literals like "team:payments" or f"user_id:{uid}"
TAG = re.compile(r"f?[\"']([a-z0-9_.\-]+):([^\"']*)[\"']")
base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
diff = subprocess.run(["git", "diff", "-U0", f"{base}...HEAD", "--", "*.py"],
capture_output=True, text=True, check=True).stdout
problems, path = [], None
for line in diff.splitlines():
if line.startswith("+++ "):
path = line[6:] # strip "+++ b/"
continue
# Only inspect added lines that contain a metric call (single-line calls only).
if not line.startswith("+") or not METRIC_CALL.search(line):
continue
for key, value in TAG.findall(line):
if key in UNBOUNDED:
problems.append(f"{path}: '{key}' is usually unbounded")
elif "{" in value:
problems.append(f"{path}: '{key}' uses an interpolated value")
elif key not in ALLOWED:
problems.append(f"{path}: '{key}' is not in the tag allowlist")
for p in problems:
print(p)
if problems:
print("Map values to a bounded set or justify the exception in review.")
sys.exit(1 if problems else 0)
A minimal CI script that scans newly added DogStatsD-style metric calls in a branch diff and fails the build when a tag key is known to be unbounded, uses an interpolated value, or is missing from the team's allowlist.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to reduce datadog costs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.