How is SaaS observability pricing structured, and where do bill surprises come from?

SaaS observability vendors, Datadog included, bill through several independent meters. Each one grows with a different property of your system. A comparison is only fair when you map each meter to the self-hosted resource that would replace it.

Meter What it counts What makes it grow Self-hosted equivalent
Infrastructure hosts / containers Monitored hosts or container units Fleet size, autoscaling peaks Agent overhead, scrape capacity
APM-instrumented hosts Hosts running a tracing agent Services you instrument Trace backend compute
Custom metrics Unique metric + tag combinations Label cardinality Prometheus memory and storage
Log ingestion Volume shipped to the vendor Log verbosity, traffic Ingest pipeline, object storage
Log indexing / retention Volume made searchable, and for how long Retention policy, query needs Index storage, query nodes
Trace spans Spans ingested or retained Request rate, sampling rate Trace storage

The per-host line is predictable because headcount of machines changes slowly and deliberately. Bill surprises usually come from the usage-scaled meters. Someone adds a user_id tag, a debug log level ships to production, or tracing sampling goes to 100% during an incident. None of these changes the host count, but each one can multiply a meter overnight.

Negotiated contracts move these numbers considerably. Committed-use discounts, overage rates and bundled allowances can push your effective price well above or below list. Secondhand "we paid X" figures are therefore close to useless. Price from your own contract and the vendor's current pricing page.

The real decision is not "Datadog vs free software." It is metered usage weighed against three things you take on when you self-host: infrastructure, engineer time, and operational risk.

Why does metric cardinality drive cost on both Datadog and self-hosted Prometheus?

A time series is one unique combination of metric name and label values. Every label you add multiplies the possible series count, so cost tracks the product of label cardinalities, not the number of metrics you define.

Illustrative estimate (assumed numbers, not a benchmark). Take http_requests_total with method (5 values), status (6), path (200), and pod (50):

series ≤ 5 × 6 × 200 × 50 = 300,000

That figure is an upper bound, because only label combinations that actually occur are stored. Pod churn makes it worse. Every deploy creates new pod values, and the old series persist until retention expires them. Adding user_id with 10,000 active users multiplies the bound by 10,000.

The cost of those series depends on where they land:

  • Datadog counts custom metrics by unique metric-and-tag combinations, so each series can become a billed unit.
  • Prometheus keeps active series in memory. Memory grows roughly with series count, the TSDB grows on disk, and any query that touches many series slows down.

Measure before you model cost:

  • TSDB status endpoint. /api/v1/status/tsdb reports the head block's series count. It also lists the metric names and label names with the most series, and the labels with the most distinct values.
  • Series per metric. topk(20, count by (__name__)({__name__=~".+"})) ranks metric names by series count.
  • Values per label. count(count by (path)(http_requests_total)) shows how many distinct values one label contributes to one metric.

The worst offenders tend to share a few traits:

  • Labels holding unbounded values, such as IDs, raw URLs, or error messages.
  • Histograms, where the bucket count multiplies every other label.
  • Labels whose values change on every deploy, such as pod.
#!/usr/bin/env bash
# Measure series cardinality on a Prometheus server before modeling cost.
# Requires curl and jq. PROM points at your server (placeholder below).
set -euo pipefail
PROM="${PROM:-"

# 1. Head block totals plus Prometheus's own top-N cardinality lists.
TSDB=$(curl -s "$PROM/api/v1/status/tsdb")
echo "== Active series in head block"
echo "$TSDB" | jq '.data.headStats.numSeries'

echo "== Metric names with the most series"
echo "$TSDB" | jq -r '.data.seriesCountByMetricName[] | "\(.value)\t\(.name)"'

echo "== Labels with the most distinct values (unbounded IDs show up here)"
echo "$TSDB" | jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"'

# 2. Run a PromQL instant query and print value<TAB>labels.
q() {
  curl -s -G "$PROM/api/v1/query" --data-urlencode "query=$1" \
    | jq -r '.data.result[] | "\(.value[1])\t\(.metric | tostring)"'
}

# 3. Rank metric names by series count. This query is expensive on large
#    servers, so run it off-peak.
echo "== Top 20 metrics by series (PromQL)"
q 'topk(20, count by (__name__)({__name__=~".+"}))'

# 4. Count the distinct values one label adds to one metric.
echo "== Distinct path values on http_requests_total"
q 'count(count by (path)(http_requests_total))'

# 5. Find which pods carry the most series for that metric. Churn from
#    deploys appears as many short-lived pod values.
echo "== Series per pod for http_requests_total"
q 'topk(10, count by (pod)(http_requests_total))'

A bash script that pulls head-block series counts and the top cardinality lists from Prometheus's /api/v1/status/tsdb endpoint, then uses PromQL count queries to rank metrics by series and find the labels and pods that contribute the most.

What does a self-hosted Grafana stack contain and how does it work?

Applications and hosts send telemetry to OpenTelemetry collectors, usually an agent on each node plus a gateway tier. Collectors batch, drop noisy attributes, sample traces, and route each signal to its backend. Prometheus scrapes targets directly or receives remote-write. For retention beyond one node's disk, data goes to a horizontally scalable, Prometheus-compatible store that keeps recent data on local disk and ships compacted blocks to object storage. Loki indexes only log labels and stores compressed chunks in object storage. Tempo writes traces to object storage and is built around lookup by trace ID. Grafana queries all three, and Alertmanager deduplicates, groups and routes alerts from the rule evaluators.

Component Role Main cost drivers
OTel collectors Ingest, process, route Compute, cross-AZ traffic to gateways
Prometheus / long-term store Metrics Memory for active series, block storage, object storage
Loki Logs Ingester compute, object storage, query compute
Tempo Traces Object storage, ingest compute
Grafana Dashboards Small compute, metadata database backups
Alertmanager Alert routing Minimal compute, clustered replicas

Replicating ingesters across availability zones protects data. It also means each write crosses zones at least once, and on some clouds that transfer fee rivals the storage bill.

Object storage makes long retention cheap, but reads are slow. A query over 90 days of metrics, or a Loki filter that must decompress every chunk because the text you want isn't a label, can take minutes. An indexed SaaS search over the same data often returns in seconds. That gap is most noticeable during an incident, when you need answers quickly.

+--------------+              +----------------+
| Apps / hosts |              | Scrape targets |
+------+-------+              +-------+--------+
       | OTLP                         ^
       v                              | scrape
+------+-------+                      |
| OTel agent   |  (each node)         |
+------+-------+                      |
       | batch, filter, sample        |
       | cross-AZ traffic $           |
       v                              |
+--------------+   remote-write+------+-------+
| OTel gateway +-------------->| Prometheus   +--alerts--+
+---+------+---+               +------+-------+          |
    |      |                          | blocks           v
    |logs  |traces                    v           +------+-------+
    v      v                   +--------------+   | Alertmanager |
+------+ +-------+             | Long-term    |   | dedupe,group |
| Loki | | Tempo |             | Prom-compat  |   | and route    |
+--+---+ +---+---+             | metrics store|   +--------------+
   |         |                 +------+-------+
   | chunks  | traces                 | compacted blocks
   v         v                        v
+--------------------------------------------------+
| Object storage: cheap long retention; slow reads |
| over wide time ranges and unindexed log text     |
+--------------------------------------------------+
+--------------------------------------------------+
| Grafana: queries Loki, Tempo, metrics store      |
+--------------------------------------------------+

OpenTelemetry collectors route metrics, logs and traces to Prometheus, Loki and Tempo. All three back onto object storage, and Grafana queries each backend while Alertmanager routes alerts; $ marks cross-AZ traffic as a cost driver.

How do you build a Datadog vs Grafana cost model with the same units?

Comparing a Datadog invoice to a Prometheus server bill only works if both sides are expressed in the same units. Use four:

  • Active series
  • GB of logs ingested per day
  • Retention days
  • Spans per second

Price the SaaS side from your own contract, including committed-use discounts and overage rates. List prices won't match what you actually pay. Price the self-hosted side with this formula:

Monthly self-hosted cost = infrastructure + (FTEs × fully loaded annual cost ÷ 12)

Then add a one-time migration cost:

Migration = (engineer-months × monthly loaded cost) + parallel-run months × self-hosted infra

Engineer-months cover rebuilding dashboards and alerts, re-instrumentation, and retraining. The parallel-run term is there because you keep paying the vendor while both systems run side by side.

Illustrative estimate (assumed numbers, not a benchmark)

Assumptions:

  • Fully loaded engineer cost of $200,000 per year.
  • Six engineer-months of migration work.
  • Two months of parallel running.
  • At 5× scale, migration doubles to 12 engineer-months.
Input Current scale 5× scale
Active series 500k 2.5M
Logs ingested/day 200 GB 1 TB
Retention 15 days 15 days
Spans/sec 2,000 10,000
SaaS (assumed contract) $40,000/mo $200,000/mo
Self-hosted infra $12,000/mo $50,000/mo
Staff 1.5 FTE = $25,000/mo 2.5 FTE ≈ $41,700/mo
Self-hosted total $37,000/mo ≈ $91,700/mo
Migration (one-time) ≈ $124,000 ≈ $300,000
Payback ≈ 41 months ≈ 3 months

At the current scale, a $3,000 monthly saving takes more than three years to repay the migration. That is long enough for your requirements or your contract to change first.

At 5× scale, the conclusion reverses. This happens because staffing grows more slowly than usage.

There is no universal crossover point. It depends on your negotiated rates, your cardinality, your retention needs, and how much of an engineer's time the stack really consumes.

# Illustrative estimate: every input is an assumption, not a measured result.
LOADED_ANNUAL = 200_000          # fully loaded cost per engineer, $/year
MONTHLY_LOADED = LOADED_ANNUAL / 12

scenarios = {
    # Units are recorded so both sides describe the same workload.
    "current": dict(series=500_000, log_gb_day=200, retention_days=15, spans_s=2_000,
                    saas=40_000, infra=12_000, fte=1.5,
                    migration_eng_months=6, parallel_months=2),
    "5x": dict(series=2_500_000, log_gb_day=1_000, retention_days=15, spans_s=10_000,
               saas=200_000, infra=50_000, fte=2.5,
               migration_eng_months=12, parallel_months=2),
}

def model(s):
    # SaaS figure comes from your own contract, not the list price.
    staff = s["fte"] * MONTHLY_LOADED
    self_hosted = s["infra"] + staff
    # Parallel run: you keep paying the vendor (already in the saas baseline)
    # while also running the new self-hosted infrastructure.
    migration = (s["migration_eng_months"] * MONTHLY_LOADED
                 + s["parallel_months"] * s["infra"])
    saving = s["saas"] - self_hosted
    # No monthly saving means the migration never pays back.
    payback = migration / saving if saving > 0 else float("inf")
    return self_hosted, migration, saving, payback

for name, s in scenarios.items():
    total, migration, saving, payback = model(s)
    print(f"{name:8} self-hosted=${total:,.0f}/mo  migration=${migration:,.0f}  "
          f"saving=${saving:,.0f}/mo  payback={payback:.1f} months")

# Output with these assumptions:
# current  self-hosted=$37,000/mo  migration=$124,000  saving=$3,000/mo  payback=41.3 months
# 5x       self-hosted=$91,667/mo  migration=$300,000  saving=$108,333/mo  payback=2.8 months

A small Python cost model that applies the section's formulas to the assumed inputs and reproduces the illustrative payback figures for both scales; replace every input with numbers from your own contract and team.

How much engineer time does a self-hosted observability stack really cost?

The work continues even when nothing is broken. A typical stack runs Prometheus, a long-term metrics store, a log store, a trace store and Grafana. Each component has its own release cycle and breaking changes. Ingestion paths need resharding as series counts grow. Slow dashboards need query and recording-rule tuning. The monitoring system also needs its own meta-monitoring: a small, separate setup that pages someone when the main stack stops scraping or ingesting. Someone has to be on call for all of it.

Estimate each task in hours per month:

FTE fraction = (sum of monthly task hours) ÷ (working hours per month)

Illustrative estimate (assumed numbers, not a benchmark), assuming 160 working hours per month and a large environment with about 20× the ingest of the small one:

Task Small (hrs/month) Large (hrs/month)
Multi-component upgrades 6 16
Capacity planning 2 8
On-call and incidents 6 20
Scaling ingestion 1 12
Query tuning 2 12
Meta-monitoring 1 4
Total / FTE 18 / 0.11 72 / 0.45

In this example, usage grows 20× while hours grow 4×. That gap comes from fixed overhead. Upgrades, on-call rotations and meta-monitoring cost roughly the same whether you ingest a little or a lot. At small scale, that 0.11 FTE multiplied by a loaded salary can exceed the entire SaaS bill. Those hours also come out of feature work.

The bigger risk is people. Operational knowledge tends to sit with one or two engineers. When they leave, the stack becomes unowned and upgrades stall. Version drift then makes the eventual upgrade larger and riskier. Plan for at least two people who can operate every component, not one.

How can you cut observability costs before switching vendors?

Whichever vendor you choose, cost tracks volume, so reduce volume first. These levers lower a SaaS bill and shrink the storage a self-hosted stack has to run.

Metrics: cut cardinality where data enters

Drop or rewrite labels before they are stored. In Prometheus, metric_relabel_configs runs after the scrape and before ingestion. The labeldrop action removes a label such as pod_ip or request_id, and the drop action discards whole series you never query. For expensive dashboard queries, use recording rules. They precompute aggregates such as per-service request rate, so you can keep the raw series on short retention. In an OpenTelemetry Collector, the attributes and filter processors do the same job before data reaches any backend.

Logs: filter by level, split by purpose

Drop DEBUG logs at the collector unless a team temporarily enables them, and sample repetitive INFO lines. Route debug and application logs to an index with a few days of retention. Send audit logs to a separate long-retention store. You pay for retention per byte, so mixing the two means paying audit-grade retention on debug noise.

Traces: sample on purpose

  • Head sampling (probabilistic_sampler) is cheap but blind: it decides before knowing how the request turned out.
  • Tail sampling (tail_sampling) keeps all errors and slow traces and samples the rest. It needs more Collector memory, because spans are buffered until the decision is made.

Attribution and guardrails

Tag all telemetry with team and service, then report volume per team every month. Alert when ingest grows faster than request volume. Together these show which team's change caused a jump, and they turn cost into a number each team owns rather than a surprise at invoice time.

# OpenTelemetry Collector (contrib): cut volume before it reaches any backend
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  memory_limiter:          # tail sampling buffers spans, so cap memory first
    check_interval: 1s
    limit_mib: 1024
  resource/team:           # attribution: every signal carries its owning team
    attributes:
      - {key: team, value: payments, action: upsert}
  attributes/strip:        # drop per-request IDs that explode cardinality
    actions:
      - {key: request_id, action: delete}
  filter/drop-debug:       # drop TRACE/DEBUG logs; keep records with no severity set
    error_mode: ignore
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_INFO and severity_number != SEVERITY_NUMBER_UNSPECIFIED'
  tail_sampling:           # keep all errors and slow traces, 10% of the rest
    decision_wait: 10s
    policies:
      - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
      - {name: slow, type: latency, latency: {threshold_ms: 500}}
      - {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 10}}
  batch: {}
exporters:
  otlp:
    endpoint: telemetry-backend.example.internal:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource/team, attributes/strip, tail_sampling, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource/team, attributes/strip, filter/drop-debug, batch]
      exporters: [otlp]

An OpenTelemetry Collector pipeline tags all telemetry with a team, strips a high-cardinality attribute, drops debug logs, and tail-samples traces so that errors and slow requests are always kept, all before the data reaches any vendor.

What are the alternatives to choosing purely Datadog or purely self-hosted Grafana?

Most teams end up somewhere between the two extremes, and the options combine well.

Instrument with OpenTelemetry first. If services emit OpenTelemetry (OTel) traces, metrics and logs through a collector, the backend becomes a routing decision. Vendor-specific agents and SDKs tie dashboards, alerts and code to one backend. OTel keeps that coupling in the collector config, so a hybrid setup or a later migration does not require touching application code. The hedge pays off as soon as you have more than a handful of services, because the cost of re-instrumenting grows with the size of the codebase.

Option Where it tends to break even Main risk
OTel everywhere, any backend Almost always; cost is mostly up front Vendor-specific features need extra setup
Selected high-value telemetry to SaaS Expensive data is a small share of the volume Two tools, split context during incidents
Managed Grafana-compatible backend Usage is too large for SaaS pricing but too small to justify an ops team Usage pricing comes back at scale
Long-term metrics in object storage, short-term data elsewhere Retention needs are long and queries on old data are rare Slow queries on historical data

The usual split is to send production traces from critical services to SaaS and keep high-volume infrastructure metrics and debug logs on cheaper storage. Before you split, check which SaaS features you actually depend on. Open-source tools have no easy equivalent for polished APM correlation, continuous profiling, real user monitoring (RUM), security products or turnkey integrations. Rebuilding those yourself often costs more than the bill you are trying to cut.

When should you self-host, and when should you avoid it?

So before deciding, gather five inputs: current monthly volume per signal (metric series, log GB, spans), the growth rate of each, whether a platform team has spare capacity, any compliance or data-residency constraints, and the product features people actually open during a normal week. The last one matters most. If on-call engineers mostly look at dashboards and alerts, a Grafana stack replaces that well. If they rely on trace-to-log correlation and profiling during incidents, you are rebuilding a product.

Factor Stay on Datadog Self-host Grafana stack
Volume and growth Moderate and flat Large and compounding
Platform capacity No spare headcount Team already runs stateful infrastructure
Compliance Vendor regions meet requirements Data must stay in your own network or region
Features used APM, RUM, profiling, security Metrics, logs, basic tracing
Incident tolerance Cannot accept a blind spot Can run an independent fallback

Also count the migration effort. Agents, client libraries, custom tags and alert definitions all need re-instrumentation and validation. Plan to run both systems in parallel for weeks, which means paying twice during that period.

The risk people underestimate is shared failure. If your observability stack runs on the same cluster and network as production, the outage you most need to debug can also take down your view of it. Self-hosting is only safe when the stack is isolated from production and has its own external health check.

Teams with small, steady usage and no platform slack should stay on SaaS and cut cardinality instead. Self-hosting fits teams whose bill grows faster than headcount and who already operate databases reliably.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to datadog vs grafana cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.