What drives trace sampling cost, and which stage does each technique reduce?

Trace sampling is the decision to keep or drop whole traces before they reach long-term storage. The goal is to pay for representative and interesting traces, not every request. Observability costs build up in stages, and each stage has its own cost driver:

  • In the service: CPU and memory to create, batch and serialize spans, plus export calls.
  • Network: egress from services to Collectors and from Collectors to the backend, which matters most across zones or regions.
  • Collector: compute to process spans, and memory to hold them while waiting to decide.
  • Backend ingest: usually priced per span or per GB.
  • Storage: grows with retention period and index cardinality.
  • Query: cost and latency rise with the volume scanned.
Technique Service overhead Network Collector Ingest / storage / query
Head sampling (in SDK) Reduced: dropped spans are not recorded or exported Reduced Reduced Reduced
Tail sampling (in Collector) Unchanged Unchanged up to the Collector Increased: complete traces are buffered Reduced
Span-derived metrics Unchanged Unchanged Slightly increased Indirect: allows lower trace sample rates
Instrumentation reduction Reduced Reduced Reduced Reduced

The table shows the main point of this article: where the keep/drop decision happens matters more than the percentage you choose. A 10% head sample and a 10% tail sample store the same volume, but they cost and reveal different things.

Head sampling decides before a request finishes, so it saves money at every stage. However, it does not know which traces will contain errors or slow spans. Tail sampling sees the full trace and can keep the interesting ones. In return, you pay full collection costs up to the Collector, plus the memory needed to hold traces until each decision is made.

How does head sampling work, and when should you avoid relying on it?

Head sampling makes the keep-or-drop decision when the root span starts. In the OpenTelemetry SDK, the usual configuration is a parent-based sampler wrapping a trace-ID-ratio sampler. For a root span, the ratio sampler compares a value derived from the trace ID against the configured probability. For every other span, the parent-based wrapper copies the parent's decision.

That decision travels with the request. The sampled flag in the W3C traceparent header tells each downstream service whether the trace was kept. Each service honours the flag rather than rolling its own dice. The result is that traces are either complete or absent, never missing a hop in the middle.

The savings start inside the process. Unsampled spans are created as lightweight non-recording spans, so the SDK skips attribute collection, batching, serialisation and export. Network egress, Collector processing and backend storage all shrink by the same ratio.

The weakness is timing. The decision happens before the request has produced an error or a slow response, so it cannot favour either. At a 5% rate you keep roughly one in twenty failed checkouts and one in twenty p99 outliers, the same as healthy requests. A failure that occurs a few times an hour may not appear at all. No tail policy or backend rule can bring it back, because those spans were never exported.

Head sampling suits high-volume, homogeneous services, such as health-check-heavy APIs or uniform read paths, where per-span SDK overhead and network cost dominate and individual requests are interchangeable.

# Head sampling: keep ~5% of traces, decided once when the root span starts.
# Equivalent env config: OTEL_TRACES_SAMPLER=parentbased_traceidratio
#                        OTEL_TRACES_SAMPLER_ARG=0.05
from opentelemetry import trace
from opentelemetry.propagate import inject
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Root spans: compare a value derived from the trace ID against 0.05.
# Child spans (local or remote parent): copy the parent's sampled flag.
sampler = ParentBased(root=TraceIdRatioBased(0.05))

provider = TracerProvider(sampler=sampler)
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("checkout-service")

with tracer.start_as_current_span("POST /checkout") as span:
    # False for ~95% of requests: a non-recording span, so attributes,
    # batching, serialisation and export are all skipped in-process.
    print("recording:", span.is_recording())

    # The decision is already fixed here, before we know if this request
    # will fail or be slow. Errors recorded on an unsampled span are lost.
    span.set_attribute("cart.items", 3)  # no-op when not recording

    # Downstream calls carry the decision in the W3C traceparent header:
    # trailing flags "01" = sampled, "00" = not sampled. Services using
    # ParentBased honour it, so traces arrive complete or not at all.
    headers = {}
    inject(headers)
    print("traceparent:", headers.get("traceparent"))

Configures a parent-based 5% trace-ID-ratio sampler in the OpenTelemetry Python SDK and shows that unsampled spans are non-recording while the keep-or-drop decision propagates downstream via the traceparent sampled flag.

How does tail sampling in the OpenTelemetry Collector work?

The tail sampling processor, tail_sampling in the Collector contrib distribution, makes its decision after a trace completes. It holds spans in memory, grouped by trace ID, for a decision_wait period. When that window closes, it evaluates the whole trace against a list of policies. If any policy votes to sample, the processor keeps every span in the trace. Otherwise it drops them all. Deciding late is what lets you keep traces based on outcomes the root span cannot know at start time, such as a downstream failure or a slow database call.

A typical policy set, matching the configuration example that follows:

  • status_code keeps traces containing an ERROR span, because failures are the traces you debug most.
  • latency keeps traces whose duration exceeds a threshold, such as your p99 SLO target, to capture slow paths.
  • string_attribute keeps traces that match specific values, such as a premium tenant ID or a newly deployed route.
  • probabilistic keeps a small baseline of everything else, so you still have normal traces for comparison.
  • and combining status_code with rate_limiting caps error traces in spans per second, which protects the backend.

That rate cap has a cost. During an error storm, the cap will drop error traces at exactly the moment you most need them. Set it from backend capacity, and alert when it saturates.

Tail sampling only reduces backend ingest and storage. The SDK still creates and exports every span, the network still carries them, and the Collector still buffers and evaluates them.

receivers:
  otlp:
    protocols:
      grpc: {}
processors:
  tail_sampling:
    decision_wait: 10s     # hold spans per trace ID this long, then decide on the whole trace
    num_traces: 50000      # traces buffered in Collector memory; you still pay this RAM/CPU
    expected_new_traces_per_sec: 1000
    policies:              # any policy voting "sample" keeps every span in the trace
      # Errors, capped. No standalone status_code policy, or it would bypass the cap.
      # In an error storm, errors above the cap are dropped. Size it from backend capacity.
      - name: errors-capped
        type: and
        and:
          and_sub_policy:
            - name: has-error
              type: status_code
              status_code: {status_codes: [ERROR]}
            - name: error-cap
              type: rate_limiting
              rate_limiting: {spans_per_second: 500}
      - name: slow           # e.g. at or above the p99 SLO target
        type: latency
        latency: {threshold_ms: 800}
      - name: premium-tenant
        type: string_attribute
        string_attribute: {key: tenant.id, values: [tenant-premium-01]}
      - name: baseline       # small sample of normal traffic for comparison
        type: probabilistic
        probabilistic: {sampling_percentage: 5}
  batch: {}
exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]   # savings start here: backend ingest and storage only
      exporters: [otlp]

A Collector traces pipeline whose tail_sampling processor buffers whole traces for 10s. It then keeps rate-capped error traces, slow traces, one premium tenant's traces and a 5% baseline, and drops the rest before export.

What architecture does tail sampling require at scale?

A tail sampler can only judge a trace it sees whole. If one Collector instance receives the root span and another receives the failing database call, neither has enough of the trace to apply an error or latency policy correctly. Every span with a given trace ID must therefore land on the same instance.

The standard layout uses two tiers. The first is a stateless layer of Collectors running the load-balancing exporter. It hashes each span's trace ID and forwards the span to one member of a second, stateful tier. That second tier runs the tail sampling processor. It holds spans in memory until the decision wait expires, evaluates its policies, and then forwards or drops the trace.

Size the stateful tier's memory for the buffering window:

buffer ≈ decision wait time × span throughput × average span size

As an illustrative estimate, assume a 30-second wait, 50,000 spans per second and 1 KB per span. That works out to about 1.5 GB of buffered spans before counting processing overhead.

The failure modes cluster together:

  • Memory pressure. If the buffer outgrows its limit, the processor drops traces or the process is killed.
  • Traffic spikes. Spans per second rise during a spike while the wait time stays fixed, so the buffer grows in proportion.
  • Short decision windows. When a trace runs longer than the wait, its late spans arrive after the decision and get dropped or end up as fragments.
  • Rebalancing. When you scale the sampling tier, the hash ring changes and in-flight traces get split across instances.
  • The extra hop. The routing tier adds latency, network cost and another component that can fail.

All of these get worse under heavy load. Incidents bring traffic spikes, long traces and autoscaling events, so the pipeline is most likely to degrade during the outages you most need traces for.

 +-----------+   +-----------+   +-----------+
 | Service A |   | Service B |   | Service C |   spans of one trace
 +-----+-----+   +-----+-----+   +-----+-----+   arrive at any node
       |               |               |
       v               v               v
 +-------------------------------------------------------------+
 | Tier 1: Collectors with load-balancing exporter (stateless) |
 | route each span by hash(trace ID) -> one sampler            |
 +-------+---------------------+---------------------+---------+
         |                     |                     |
         v                     v                     v
 +---------------+     +---------------+     +---------------+  Tier 2
 | Sampler A     |     | Sampler B     |     | Sampler C     |  (stateful)
 | tail sampling |     | tail sampling |     | tail sampling |
 | span buffer   |     | span buffer   |     | span buffer   |
 +-------+-------+     +-------+-------+     +-------+-------+
         |                     |                     |
         +---------------------+---------------------+
                               |
                               v  kept traces only
                     +-------------------+
                     | tracing backend   |
                     +-------------------+

 buffer per sampler ~= decision wait x span throughput x avg span size
 scaling Tier 2 changes the hash ring; in-flight traces can split

A stateless load-balancing tier hashes each span's trace ID so that every span of a trace reaches the same stateful tail-sampling Collector, which buffers spans for the decision wait before keeping or dropping the trace.

Illustrative estimate: no sampling vs head, tail and layered sampling

Pipeline risk is hard to price, but the ingest bill is easy to model. The figures below come from assumed inputs so you can substitute your own.

Illustrative estimate (assumed numbers, not a benchmark)

  • 2,000 requests/s, 30-day month
  • 30 spans per trace, 500 bytes per span
  • 1% of traces are errors (e), 2% are slow (s), with no overlap
  • $0.30 per GB ingested
  • Tail baseline keep rate (b) of 5%

Formulas

  • Monthly GB = req/s × 86,400 × 30 × spans × bytes × kept fraction ÷ 10⁹
  • Tail kept fraction = e + s + b(1 − e − s)

With these inputs, unsampled traffic is 77,760 GB per month. That works out to 1.73M error traces and 3.46M slow traces per day.

Setup Kept fraction GB/month Ingest $/month Errors kept/day Slow kept/day
No sampling 100% 77,760 23,328 1.73M 3.46M
10% head 10% 7,776 2,333 173K 346K
Tail (all errors and slow, plus 5%) 7.85% 6,104 1,831 1.73M 3.46M
10% head, then tail 0.785% 610 183 173K 346K

The ingest column hides where each setup saves money:

  • Head sampling cuts every downstream stage: SDK export, network, collector CPU and backend ingest. It drops 90% of failures blindly.
  • Tail sampling cuts only backend ingest and storage. The collector tier still receives, buffers and evaluates 100% of spans, so its compute and memory cost is not in this bill.
  • Head-then-tail makes the collector tier roughly ten times smaller. Error coverage is capped at the head rate. You can raise the head rate to recover coverage, which costs more collector capacity.

How do span-derived metrics keep dashboards accurate at any sample rate?

Sampling decides which traces you keep. It should not decide what your dashboards say. The fix is to compute RED metrics (request rate, error count, duration histograms) from every span in the Collector, before any sampler runs. In the OpenTelemetry Collector, the span metrics connector does this. Place it in a pipeline that receives unsampled spans, and export its output to your metrics backend. Your SLOs then rest on complete data, and trace retention becomes a separate cost decision.

The alternatives are weaker:

  • Extrapolating from sampled traces only works if each span carries the probability it was kept with, so you can weight it by the inverse. If you sample at 10%, each kept span counts as ten. Without that recorded probability, you have counts you cannot scale back up.
  • Mixing samplers breaks extrapolation entirely. A tail policy that keeps every error but 5% of successes inflates the apparent error rate. It also drags p99 toward the slow tail. If services apply head rates inconsistently, you get partial traces and per-service counts on different scales.
  • Separate metrics instrumentation is accurate, but it duplicates work. Its labels can also drift from your span attributes, which makes correlating a metric spike with traces harder.

Watch cardinality. Every dimension you add to span metrics multiplies the number of time series. Grouping by user ID, full URL path or request ID can create millions of series. That can cost more than the traces you dropped. Use route templates, status codes and service names instead.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [5ms, 25ms, 100ms, 250ms, 1s, 5s]
    # Low-cardinality dimensions only: route templates and status codes.
    # Never add user IDs, request IDs or full URLs here.
    dimensions:
      - name: http.route
      - name: http.response.status_code

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  otlp/traces:
    endpoint: traces-backend.example.internal:4317
  otlp/metrics:
    endpoint: metrics-backend.example.internal:4317

service:
  pipelines:
    traces/all:          # every span, no sampler: feeds RED metrics
      receivers: [otlp]
      exporters: [spanmetrics]
    traces/sampled:      # same spans, tail-sampled for retention
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp/traces]
    metrics/red:
      receivers: [spanmetrics]
      exporters: [otlp/metrics]

An OpenTelemetry Collector config in which the span metrics connector computes RED metrics from every span in an unsampled pipeline. A separate pipeline tail-samples the same spans for storage, so dashboard accuracy does not depend on how many traces are kept.

How should you roll out trace sampling without losing signal?

Order the work so each step shrinks the problem for the next one.

  1. Measure span volume per service and operation. A handful of endpoints usually dominate. You cannot set sensible rates without knowing which ones.
  2. Drop noise at the source. Health checks, readiness probes and chatty internal spans (per-row database calls, cache lookups inside loops) cost money and carry almost no diagnostic value.
  3. Add span-derived metrics so dashboards and alerts stop depending on how many traces you keep.
  4. Introduce tail policies for errors, slow requests and critical routes.
  5. Only then lower head sampling rates.

Following this order means you tune once. If you cut head rates first, you end up tuning rates against volume you later delete, and you break dashboards you then have to rebuild.

Define "keeping signal" as tests you run after every change. During an incident, can you still find a trace for:

  • every error class,
  • p99 latency outliers,
  • a specific customer or request ID?

If any of these answers is no, revert the last step.

On-demand overrides fill the gaps. Examples are a debug header that a gateway turns into a forced-sample decision, or a tail policy that keeps every trace with a debug attribute or a flagged tenant. Overrides need governance:

  • Restrict who can set them, or strip them at the edge so external clients cannot inflate your bill.
  • Give them an expiry.
  • Log their use.

Smaller teams with modest volume can often stop after step 3 and use a fixed head rate. Tail sampling's stateful collectors are worth running only once trace storage becomes a meaningful line item.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to trace sampling cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.