How do you estimate distributed tracing costs?

Tracing cost is driven by how many spans you keep and how many bytes each one occupies. Request rate is set by your traffic, so the model is:

  • Stored spans/s = requests/s × spans per request × sampling rate
  • Ingest bytes/day = stored spans/s × average bytes per span × 86,400
  • Stored bytes = ingest bytes/day × retention days

Whether your vendor bills on spans, bytes, or hosts (covered next), these quantities sit underneath the price.

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs: 2,000 requests/s, 40 spans per request, 10% head sampling, 1,000 bytes per span, 15-day retention.

Scenario Spans/request Sample rate Stored spans/s Ingest/day Stored (15 days) Traces kept/s
Baseline 40 10% 8,000 691 GB ~10.4 TB 200
Halve sample rate 40 5% 4,000 346 GB ~5.2 TB 100
Halve spans/request 20 10% 4,000 346 GB ~5.2 TB 200

The two cuts produce the same bill, but they keep different amounts of evidence. Halving the sample rate throws away half of your traces. That leaves fewer examples of rare errors and slow outliers. Halving spans per request keeps every sampled trace and removes the spans that added the least information, such as per-row database calls or chatty internal helpers.

Sampling rate, spans per request, and bytes per span multiply together. Cutting any one of them by half halves the cost. Most teams only adjust sampling because it is a single setting. This article covers all three levers, plus the cases where sampling is still the right tool.

Which APM pricing dimension is your bill actually metered on?

The three-lever formula only predicts your invoice if your vendor meters on the same quantity. APM pricing usually uses one or more of the following dimensions, and each one responds to a different lever.

Metered dimension Levers that reduce it Levers that do not
Ingested spans Sampling rate, spans per request Span size
Ingested bytes All three: sampling, span count, span size None
Indexed or retained spans Retention filters, indexing rules Ingest-side cuts, if filters already select a fixed subset
Hosts or agents Fewer or larger hosts Sampling, span count, span size
Retention period Shorter retention Most ingest optimizations, unless also billed by volume

Mismatches are common. If a vendor bills on indexed spans and your retention filters already keep only errors and slow traces, lowering head sampling cuts ingest but may leave the indexed count almost unchanged. Stripping large attributes saves money on a bytes-metered plan but does nothing on a span-count plan. On a per-host plan, none of the tracing levers matter much. Consolidating services onto fewer nodes saves more there.

Self-hosting replaces the vendor meter with your own storage, compute and operational effort. Storage scales with bytes and retention. The cost that is easiest to underestimate is people. Someone has to run the collectors, size the backend, handle upgrades and respond when the backend falls behind during an incident, which is when you most need it.

Before you choose what to cut, read your contract or billing page and identify the exact metered unit. That check takes minutes and can save weeks of work on optimizations that never change the invoice.

Why does span volume per request keep growing?

Span volume grows without anyone deciding it should. Auto-instrumentation wraps every database query, cache lookup, outbound HTTP call and message publish by default, and some agents also instrument framework internals such as middleware chains and serializers. Each new client library a team adopts brings its own spans, and nobody revisits the per-request total because no single change looks expensive.

Fan-out multiplies the effect. Here is an illustrative estimate for one checkout request, with assumed inputs:

  • API gateway: 2 spans (server span plus one outbound client span)
  • Orders service: 1 server span, 4 database queries, 2 cache lookups, 3 outbound client spans = 10
  • Inventory, pricing and payment services: about 6 spans each = 18

That adds up to 30 spans. Now assume the orders service checks stock once per line item and each check emits a cache span. A 20-item cart adds 20 spans, for 50 total. The request count hasn't changed, but span volume has grown by two-thirds. Volume now depends on payload shape and call-graph depth, not only on traffic, which is why the bill can rise while request rates stay flat.

Two groups should watch this. Platform teams that own shared instrumentation libraries control the defaults, and a single default change reaches every service at once. Service owners who add manual spans control the loops: a span inside a per-item function or a retry loop scales with data size, and a load test with small fixtures won't reveal that growth.

How does head sampling work, and where does it fall short?

Head sampling makes the keep-or-drop decision once, at the root span, before any downstream work happens. The root service's sampler sets the sampled bit in the trace context, which is the trace-flags field of the W3C traceparent header (01 for sampled, 00 for not sampled). Every downstream service reads that flag and follows it. Either the whole trace is recorded or none of it is, and cost scales linearly with the rate: 1% sampling means roughly 1% of span volume.

gateway (root: sampler decides -> sampled=1)
   |-- traceparent ...-01 --> orders     (parent says 1: record)
   |                            |-- ...-01 --> inventory (record)
   |-- traceparent ...-01 --> payments   (parent says 1: record)

In the OpenTelemetry SDK, set OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.01 on every service. The ratio applies only where a trace starts. Everywhere else, the parent's flag decides.

The weakness is timing. The decision is made before the request has an outcome, so the sampler cannot tell an error from a success or a 4-second request from a 40 ms one. At 1%, you keep about 1% of errors and slow requests. A failure that occurs a few times an hour will often leave no trace at all.

Consistency is the other risk. If one service uses a plain ratio sampler, or a different rate, it makes its own independent decision. Some traces then lose a middle hop and fragment into disconnected pieces. Other traces arrive with only a single service's spans. You pay to ingest and store these spans, but they are of little use for debugging.

# docker-compose.yml (excerpt): one sampler config shared by every service
# so the root makes the decision and every downstream hop honors it.
x-otel-sampling: &otel-sampling
  # parentbased_*: if an incoming traceparent exists, follow its sampled flag.
  # traceidratio: only used when this service starts a new trace (no parent).
  OTEL_TRACES_SAMPLER: parentbased_traceidratio
  OTEL_TRACES_SAMPLER_ARG: "0.01"   # keep ~1% of traces started here
  OTEL_EXPORTER_OTLP_ENDPOINT:

services:
  gateway:            # trace root: the 1% ratio is applied here
    image: example/gateway:latest
    environment:
      <<: *otel-sampling
      OTEL_SERVICE_NAME: gateway

  orders:             # child: obeys traceparent flag 01 or 00
    image: example/orders:latest
    environment:
      <<: *otel-sampling
      OTEL_SERVICE_NAME: orders

  payments:
    image: example/payments:latest
    environment:
      <<: *otel-sampling
      OTEL_SERVICE_NAME: payments

  inventory:
    image: example/inventory:latest
    environment:
      # Anti-pattern (do not do this): a plain ratio sampler ignores the
      # parent's flag and rolls its own dice, producing partial traces.
      #   OTEL_TRACES_SAMPLER: traceidratio
      #   OTEL_TRACES_SAMPLER_ARG: "0.05"
      <<: *otel-sampling
      OTEL_SERVICE_NAME: inventory

A Compose excerpt that applies the same parent-based 1% ratio sampler to every service through shared OpenTelemetry environment variables, with the inconsistent per-service ratio sampler shown commented out as the anti-pattern.

How can you cut span volume and span size without losing coverage?

Sampling decides which traces you keep. The cheaper lever is making each kept trace smaller, which is a separate decision from how many traces you keep.

Fewer spans per request

  • Disable noisy auto-instrumentation. Libraries that wrap every cache get, DNS lookup or connection-pool checkout can produce more spans than your business logic does. Turn off the instrumentations whose spans you never open during debugging.
  • Collapse loops. A batch job that creates one span per item scales span count with input size. Wrap the whole loop in one span and record the iteration count as an attribute, such as batch.item_count. Record failures as a count too.
  • Use span events for point-in-time facts. A retry or a cache miss has no meaningful duration of its own. An event on the parent span is smaller than a child span and keeps the context.
  • Drop probe traffic. Health checks and readiness probes run constantly and carry no user context. Filter them at the instrumentation layer or in the collector.

Fewer bytes per span

Full SQL text, request bodies, long stack traces and high-cardinality attributes such as user IDs or raw URLs drive storage size. Parameterize queries, avoid recording payloads, and set limits with the standard SDK variables:

  • OTEL_ATTRIBUTE_COUNT_LIMIT
  • OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT
  • OTEL_SPAN_EVENT_COUNT_LIMIT

Review removals before you make them

The detail you remove may be exactly what an incident needs. You usually find this out only during that incident. Review each removal with on-call engineers before you ship it.

from opentelemetry import trace
from opentelemetry.sdk.trace import SpanLimits, TracerProvider

# Same limits as the env vars OTEL_ATTRIBUTE_COUNT_LIMIT=64,
# OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=1024 and OTEL_SPAN_EVENT_COUNT_LIMIT=32.
# Explicit arguments take precedence over the env vars.
limits = SpanLimits(max_attributes=64, max_attribute_length=1024, max_events=32)
trace.set_tracer_provider(TracerProvider(span_limits=limits))
tracer = trace.get_tracer("billing.batch")


def handle(item):
    ...  # business logic; deliberately creates no span of its own


def process_batch(items):
    failed = 0
    # One span for the whole batch, so span count no longer grows with input size.
    with tracer.start_as_current_span("process_batch") as span:
        span.set_attribute("batch.item_count", len(items))
        for item in items:
            try:
                handle(item)
            except Exception as exc:
                failed += 1
                # Keep only the first failure as a span event. The stack trace
                # is truncated by max_attribute_length.
                if failed == 1:
                    span.record_exception(exc)
        # Report failures as a count, not as one span or event per item.
        span.set_attribute("batch.failed_count", failed)

A Python OpenTelemetry example that replaces per-item spans with a single batch span carrying item and failure counts, and sets the SDK's attribute count, attribute length and event count limits.

When should you use per-route rates or tail sampling instead of one global rate?

A single global rate spends budget in proportion to traffic, not value. Health checks and polling endpoints dominate request counts, so they dominate kept traces. A checkout or admin route that runs a few hundred times an hour may keep almost nothing at the same rate. Per-route or per-service head rates fix this: you sample noisy endpoints at a fraction of a percent and rare, important ones near 100%. The cost is predictability. Spend now depends on the traffic mix, so a surge on a fully sampled route moves the bill. Rate tables spread across services are also harder to audit than one number.

Tail sampling at a collector tier decides after a trace completes, so it can keep every error and slow trace. To do that, it must:

  • Buffer whole traces in memory.
  • Route all spans of a trace to the same collector instance by trace ID.
  • Wait out a decision delay before keeping or dropping.

Spans that arrive after the decision window, such as async consumers or retries, miss the decision. They are typically dropped or kept as fragments, and nothing alerts you to it.

Approach Cost predictability Error capture Operational overhead Complexity
Global head High Proportional to rate Low Low
Per-route head Medium Good on routes you configured Low Medium
Tail Low to medium Strong High High

When to avoid each:

  • Global head: skip it when traffic is heavily skewed toward a few endpoints.
  • Per-route head: skip it when nobody owns the rate table.
  • Tail sampling: skip it when traces routinely outlive any practical buffer window, or when you cannot operate a stateful, trace-ID-routed collector tier.

How do you keep dashboards and alerts accurate when you sample traces?

Sampling is a cost control for trace storage, not for measurement. Request rate, error rate, and latency percentiles should come from data that saw every request. That means application metrics such as counters and histograms emitted by the service, or span-derived metrics computed in the collector before the sampling decision. If those numbers come from stored traces instead, every dashboard depends on your sampling configuration.

Counting sampled traces directly is wrong. At a 10% head rate, a panel showing 50 requests per second really represents about 500. If someone later changes the rate to 5%, the graph halves overnight and looks like a traffic drop. Multiplying by the inverse of the sample rate fixes the average, but the variance stays. Take an illustrative estimate: an endpoint handling 60 requests per minute with a 1% error rate, sampled at 1%. You would expect roughly one stored error every 100 minutes. Any single sampled error scales up to "100 errors," and most minutes show zero. An error-rate alert built on that series will flap or stay silent. Low-traffic routes and low sample rates make the problem worse.

Misleading alerts and recurring bill surprises both cost engineering hours. A fixed order of work prevents both:

  1. Confirm which dimension you are billed on: spans, bytes, or hosts.
  2. Reduce spans and bytes per request.
  3. Set per-route head sampling rates.
  4. Add tail sampling only if rare errors still go missing.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to distributed tracing costs, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.