What drives a Datadog bill, and why isn't it host count?

Datadog bills on several independent meters, and only some of them track how many machines you run. That is why two teams with similar fleet sizes can see very different invoices.

Dimension What you pay for What makes it grow
Infrastructure Each monitored host Fleet size, autoscaling peaks
APM Each host running traced services, plus span volume above the included allotment Request rate, trace sampling rate
Span retention Spans kept for search beyond the default window Retention filters, how long you keep data
Logs (ingest) Volume sent in Verbosity, debug logging left on
Logs (index) Events made searchable, priced by retention period Indexing everything instead of filtering
Custom metrics Each distinct time series (metric name plus tag combination) Tag cardinality

Host-based meters are predictable because they scale with infrastructure you already budget for. The usage-based meters are not. A custom metric tagged with customer_id or pod_name creates a new billable series for every unique value, so one metric can become thousands of series without anyone adding a host. Log indexing works the same way: ingestion is relatively cheap, but making every event searchable for 15 or 30 days multiplies cost by event count and retention period.

This article compares that bill against running the open-source LGTM stack (Loki, Grafana, Tempo, Mimir) on your own infrastructure. Teams usually start looking at alternatives after a cardinality or logging spike produces an invoice nobody forecast. A self-hosted stack still costs money, but the cost shows up as compute, storage and engineering time instead of per-series and per-event fees.

How does a self-hosted LGTM stack work, and what does it really cost to run?

Loki (logs), Mimir (metrics) and Tempo (traces) share one architecture, and Grafana sits on top as the query and dashboard layer. Agents such as the OpenTelemetry Collector run on each node and push telemetry to distributors. Distributors validate the data, apply rate limits and hash each series or stream to a set of ingesters. Ingesters hold recent data in memory, write it to a local write-ahead log, and periodically flush compressed blocks or chunks to object storage. Queriers read from ingesters for recent data and from object storage for everything older. Compactors and store-gateways handle the long tail.

That design determines where the money goes:

  • Compute: Ingesters are memory-bound, and their memory scales with active series and streams, not with raw volume. Querier compute scales with how often and how widely people search.
  • HA replication: Each write typically lands on three ingesters, so ingester memory and CPU are roughly tripled before a single query runs.
  • Cross-zone traffic: Spreading those replicas across availability zones for resilience means most writes cross zone boundaries, and cloud providers bill that traffic per GB in each direction.
  • Object storage: Storage is cheap per GB, but compaction and queries generate request charges and egress.
  • Query headroom: You size queriers for the incident, when twenty engineers run broad searches at once, not for a quiet Tuesday.

That last point carries a failure mode. An outage drives up log volume, cardinality and query load all at once. The monitoring stack is often hit hardest at the moment you need it, and if it shares a cluster or zone with the failing workload, it can go down along with it.

 +------------+ +------------+ +------------+
 | OTel Coll. | | OTel Coll. | | OTel Coll. |   one agent per node
 +-----+------+ +-----+------+ +-----+------+
       +--------------+--------------+
                      | push
                      v
       +-----------------------------+
       | Distributors                |   validate, rate-limit
       | hash each series/stream     |   to 3 ingesters
       +--------------+--------------+
                      | x3 replicas across zones:
                      | cross-zone traffic billed per GB
       +--------------+--------------+
       v              v              v
 +------------+ +------------+ +------------+
 | Ingester A | | Ingester B | | Ingester C |----------+  recent data
 | zone 1     | | zone 2     | | zone 3     |          |
 | mem + WAL  | | mem + WAL  | | mem + WAL  |          |
 +-----+------+ +-----+------+ +-----+------+          |
       |  flush blocks/chunks                          v
       v                                   +-----------------------+
 +---------------------------+             | Queriers              |
 | Object storage            |<------------| sized for incident    |
 | $/GB + requests + egress  |  old data   | query bursts          |
 +-------------+-------------+             +-----------+-----------+
               ^                                       ^
               |                                       | queries
 +-------------+-------------+             +-----------+-----------+
 | Compactor / store-gateway |             | Grafana dashboards    |
 +---------------------------+             +-----------------------+

Write and read paths of a self-hosted LGTM stack, showing where cost accrues: triple-replicated, memory-bound ingesters, cross-zone write traffic, object storage requests and egress, and queriers sized for incident-time query bursts.

How much engineering time does self-hosting add?

Running the stack yourself also moves work onto your team, and most cost comparisons leave that work out. Some of it continues for as long as you run the stack. Some of it is a one-time migration project.

Work item Recurring or one-off Why it takes time
Upgrades across Loki, Grafana, Tempo, Mimir Recurring Components release on separate schedules, and config or storage-schema changes need testing
Capacity planning Recurring Ingest and query load grow with services and cardinality, not with headcount
On-call for the stack Recurring Someone has to own the pager for the system that pages everyone else
Rebuilding dashboards, alerts, instrumentation One-off Vendor queries and agent auto-instrumentation don't port, so services move to OpenTelemetry SDKs
Double-run period One-off You pay the vendor while you validate parity on the new stack

Feature gaps add work too. Integrated APM service maps, RUM and security products have no drop-in equivalents, so you either build them, accept the gap, or keep paying for those products separately.

Loki indexes labels, not log content. That keeps ingest cheap. The cost shows up during debugging: if a request ID isn't a label, finding it means scanning every chunk in the time range. Engineers learn to narrow the time window first, which is slower in the middle of an incident.

Illustrative estimate (assumed numbers, not a benchmark)

Year-one people cost = (ongoing FTE × loaded salary) + (migration hours × hourly rate) + (double-run months × current monthly bill)

Assumed inputs:

  • 0.5 FTE ongoing at $200,000 loaded salary
  • 1,200 migration hours at $100/hour
  • 3 months of double-run against a $30,000/month bill

Year-one people cost = $100,000 + $120,000 + $90,000 = $310,000

From year two onward, only the ongoing $100,000 remains.

Where is the break-even point? A worked cost model

Illustrative estimate (assumed numbers, not a benchmark)

…ongoing operating cost remains. To turn that into a break-even point, assume this workload:

  • 200 hosts, 500 GB of logs/day with 15-day retention
  • 2 million active metric series with 13-month retention, 5,000 spans/sec
  • Current SaaS bill: $30,000/month ($360,000/year)
  • Self-hosted infrastructure (compute, object storage, egress): $8,000/month ($96,000/year)
  • Ongoing operations: 0.5 FTE at a $200,000 loaded cost ($100,000/year)

The model uses two formulas:

  • Annual savings = SaaS cost − (infrastructure + ongoing people)
  • Payback months = 12 + (year-one deficit ÷ annual savings × 12)

At baseline, year one costs $96,000 + $310,000 = $406,000. That is $46,000 more than staying on SaaS. From then on, the stack saves $164,000/year, so it pays back about 15 months after the migration starts.

Scenario (one input changed) SaaS/year Self-hosted steady/year Annual savings Payback
Baseline $360k $196k $164k ~15 months
25% negotiated SaaS discount $270k $196k $74k ~34 months
Ops needs 1 FTE, not 0.5 $360k $296k $64k ~39 months
Spans double (SaaS +$6k/mo, infra +$2k/mo) $432k $220k $212k ~12 months

The crossover depends most on two inputs: the SaaS discount and the headcount needed to run the stack. A commit contract or enterprise discount can push payback past the point where most teams would re-plan. Underestimating on-call load does the same. Growth in data volume moves payback the other way, because SaaS pricing scales with usage while self-hosted infrastructure costs rise more slowly.

Run the model with your contract terms and actual usage figures instead of list prices.

# Illustrative break-even model. All inputs are assumptions, not measurements.
# Replace them with your contract terms and actual usage.

ONE_TIME_MIGRATION = 210_000  # year-one build/migration effort (assumed)
LOADED_FTE = 200_000          # loaded cost per engineer per year

def model(saas_yr, infra_yr, ops_fte):
    ongoing_people = ops_fte * LOADED_FTE
    steady = infra_yr + ongoing_people              # self-hosted cost per year after migration
    savings = saas_yr - steady                      # annual savings
    if savings <= 0:
        return steady, savings, None                # never pays back
    year_one = infra_yr + ONE_TIME_MIGRATION + ongoing_people
    deficit = year_one - saas_yr                    # extra spend in year one vs staying on SaaS
    payback = 12 + deficit / savings * 12           # months after migration starts
    return steady, savings, payback

# Each scenario changes one input from the baseline.
scenarios = {
    "Baseline":               dict(saas_yr=360_000, infra_yr=96_000,  ops_fte=0.5),
    "25% SaaS discount":      dict(saas_yr=270_000, infra_yr=96_000,  ops_fte=0.5),
    "Ops needs 1 FTE":        dict(saas_yr=360_000, infra_yr=96_000,  ops_fte=1.0),
    "Spans double":           dict(saas_yr=432_000, infra_yr=120_000, ops_fte=0.5),
}

for name, inputs in scenarios.items():
    steady, savings, payback = model(**inputs)
    pb = f"~{payback:.0f} months" if payback is not None else "no payback"
    print(f"{name:<20} steady ${steady/1000:.0f}k/yr  "
          f"savings ${savings/1000:.0f}k/yr  payback {pb}")

A small Python script that reproduces the table's illustrative estimate and lets you change one input at a time (discount, ops headcount, data volume) to see how the payback point moves.

Why is cardinality the cost driver in both models?

Both pricing models charge for the same underlying thing: unique time series. A series is one metric name plus one specific combination of label (or tag) values, so the count is roughly the product of distinct values per label, not the sum.

Illustrative estimate (assumed numbers, not a benchmark)

Formula: series = distinct_values(label_1) × distinct_values(label_2) × … × distinct_values(label_n)

Assume a request-latency metric tagged with endpoint (50 values), status (5) and region (3): 50 × 5 × 3 = 750 series. Add a customer_id tag with 2,000 values, and it becomes 750 × 2,000 = 1,500,000 series from a single metric.

In Datadog, each of those combinations counts toward custom-metric billing. In Mimir, each active series is held in ingester memory and replicated, so the same change shows up as larger or more ingester nodes. Loki works the same way at the level of label sets: every unique set becomes its own stream. High-cardinality labels produce many small streams and chunks, which inflates index size and slows queries.

Label hygiene

  • Keep labels bounded: environment, service, region, normalized route, status class.
  • Never use user IDs, request IDs, trace IDs, raw URLs or timestamps as labels. Put them in log lines or span attributes instead.
  • Normalize paths to route templates before they become label values.
  • Drop or aggregate unused labels at the collector, before data is ingested.

Usage alerting

Alert on the growth rate of active series and log streams, not only on absolute totals, and compare against the previous deploy. Use per-tenant series and stream limits in Mimir and Loki, and usage monitors in Datadog. That way a single bad label shows up within hours as a rejected write or a page, rather than at the end of the month as an invoice or an out-of-memory ingester.

# File 1: Prometheus-style alert rules, loaded into the Mimir ruler or Prometheus.
# Alerts on growth rate versus one hour ago, so a bad deploy pages within hours
# instead of surfacing on the monthly bill.
groups:
  - name: cardinality-growth
    rules:
      - alert: MimirActiveSeriesGrowth
        # Ratio of in-memory series now vs 1h ago (replication cancels out in the ratio).
        expr: sum(cortex_ingester_memory_series) / sum(cortex_ingester_memory_series offset 1h) > 1.2
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Active series up >20% in 1h; check the most recent deploy for new labels"
      - alert: LokiStreamGrowth
        # Same idea for Loki: each unique label set is a stream.
        expr: sum(loki_ingester_memory_streams) / sum(loki_ingester_memory_streams offset 1h) > 1.2
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Loki streams up >20% in 1h; look for unbounded labels"
---
# File 2: Mimir runtime overrides. Hard per-tenant cap so excess series are rejected.
overrides:
  team-checkout:                         # placeholder tenant ID
    max_global_series_per_user: 1500000
---
# File 3: Loki runtime overrides. Hard per-tenant cap on active streams.
overrides:
  team-checkout:
    max_global_streams_per_user: 10000

Growth-rate alerts on Mimir active series and Loki streams, plus per-tenant limits. A single high-cardinality label then triggers a page or rejected writes instead of a sudden capacity or bill spike. Thresholds and tenant names are placeholders.

How can you cut observability spend before migrating?

These levers work on either backend, so apply them first. They shrink the bill you are comparing against and the system you would have to run.

Sample traces at the Collector. Head sampling is cheap but decides before the outcome is known. Tail sampling buffers whole traces, so it can keep every error and every slow request and drop most routine successes. That shifts spend toward the traces people actually open.

Drop or aggregate metrics before export. Use Collector processors to remove unused metrics and to strip high-cardinality attributes such as pod IDs or request paths. Aggregating per-pod series into per-service series cuts the series count, which is what both Datadog custom metrics and Mimir memory scale with.

Archive logs instead of indexing them. Send debug and access logs to object storage and index only what you query. If you need archived logs during an incident, you can rehydrate or rescan them.

Shorten retention on noisy data. Keep high-volume, low-value streams for days, not months.

Illustrative estimate (assumed numbers, not a benchmark): monthly indexing savings = daily log GB × fraction moved to archive × per-GB index price × 30. With 200 GB/day, 60% archived, and $0.10/GB, that is 200 × 0.6 × 0.10 × 30 = $360/month. Plug in your own values.

Less data also means fewer ingester restarts, smaller compactions, and fewer cardinality pages. That saves engineer time as well as money.

The risk is in what you remove. Slow leaks, rare race conditions, and quarterly batch failures often show up only in long retention or in the "boring" successful traces you dropped. Before you cut a data source, check whether a past postmortem relied on it.

# OpenTelemetry Collector (contrib): cut volume before export, on any backend.
receivers:
  otlp:
    protocols:
      grpc:
processors:
  # Tail sampling: wait for the whole trace, then decide on its outcome.
  tail_sampling:
    decision_wait: 10s        # buffer time per trace; raise it if you have long traces
    num_traces: 50000         # in-memory trace budget; size this to your traffic
    policies:
      - name: keep-errors     # keep every failed request
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow       # keep every request slower than 500 ms
        type: latency
        latency: {threshold_ms: 500}
      - name: some-routine    # keep 10% of other traces as a baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
  # Drop metrics nobody queries (example name; check dashboards and alerts first).
  filter/unused:
    error_mode: ignore
    metrics:
      metric:
        - 'IsMatch(name, "^http.server.request.size.*")'
  # Keep only the attributes you group by. Series that differ only in removed
  # attributes collide, so make sure your backend or an aggregation step merges them.
  transform/cardinality:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - keep_keys(attributes, ["http.route", "http.response.status_code"])
exporters:
  otlphttp:
    endpoint:    # placeholder
service:
  pipelines:
    traces:  {receivers: [otlp], processors: [tail_sampling], exporters: [otlphttp]}
    metrics: {receivers: [otlp], processors: [filter/unused, transform/cardinality], exporters: [otlphttp]}

A Collector config that keeps all error and slow traces, keeps 10% of routine traces, drops an unused metric, and trims metric attributes to cut series count before data reaches Datadog or a self-hosted Grafana stack.

Which option should your team choose, and when should you avoid self-hosting?

Score the decision against your own measured volumes. List prices and vendor calculators won't reflect your workload. Check these five inputs:

  • Monthly volume per signal. Measure log GB ingested, active metric series and spans retained, each separately. One signal usually dominates, and it may be the only one worth moving.
  • Projected growth. Model 12 to 24 months ahead. Savings that look small today grow with usage-based pricing.
  • Team size and SRE capacity. Name the people who will patch, scale and be on call for the stack itself.
  • Compliance and data residency. Some regulated data must stay in infrastructure you control.
  • Acceptable recovery time. Your monitoring can fail during the same outage it should explain. Decide how long you can tolerate that.
Factor Favors Datadog Favors self-hosted LGTM
Volume Moderate and stable Large, dominated by one signal
Platform team A few engineers, no spare capacity Dedicated observability or SRE owners
Residency Vendor regions are acceptable Data must stay in-house
Stack recovery time Must be minimal, no in-house runbooks Team already runs stateful systems reliably

Avoid self-hosting when the people cost exceeds the savings. A small platform team that spends a fraction of an engineer's year tuning compaction and object storage has often paid more than the bill it cut. That time also comes out of product reliability work.

You don't have to choose all at once. Instrument with OpenTelemetry and route through a collector you control. You can then send high-volume logs to Loki while keeping APM in Datadog, and reverse that split later without touching application code.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to datadog vs self hosted grafana cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.