OpenTelemetry Tail Sampling: A Cost Analysis for Trace Budgets

What is OpenTelemetry tail sampling?

Tail sampling decides whether to keep a trace after its spans have been collected, not when the trace starts. In OpenTelemetry it is implemented by the tail_sampling processor in the OpenTelemetry Collector. The processor buffers spans by trace ID, waits for a configured period, and then evaluates policies against the whole trace. A policy can match error status, end-to-end latency, span attributes, or a probabilistic share of traffic. If any policy matches, every span of the trace is forwarded to the backend. If none matches, every span of the trace is dropped.

Tail sampling is a way to cut trace volume and cost while still keeping the traces you most need for debugging. A fixed-ratio sampler keeps failures in the same proportion as successes. A tail sampler can keep every failing or slow trace in full and a small fraction of healthy ones.

How does it work?

At a high level, a tail-sampled pipeline has four stages:

1. Instrumented services export spans over OTLP to a collector you run. They can send everything or apply light head sampling first.
2. A routing tier groups spans by trace. A load-balancing exporter hashes each span's trace ID so that all spans of one trace reach the same sampling instance.
3. The sampling tier buffers and decides. The tail_sampling processor holds spans in memory for decision_wait, evaluates its policies with OR semantics, and then forwards or discards the whole trace.
4. The backend receives only kept traces. Ingest, storage and query charges are based on the reduced volume.

The rest of this article covers each stage in detail: what drives cost, how head and tail sampling differ, how the processor and its policies behave, how to build the two-tier topology, and how to model whether it pays off.

Where trace cost comes from and why sampling must be trace-level

Trace cost scales with two variables: how many spans you emit and how large each span is. Span count is request volume multiplied by fan-out. A single checkout request that crosses several services, each making database and cache calls, can produce dozens of spans. Span size grows with attributes, span events and recorded exceptions. A span carrying full SQL text, HTTP headers or a stack trace can be several times larger than a bare one.

Illustrative estimate (assumed numbers, not a benchmark): daily bytes = requests/day × spans/request × avg bytes/span 50,000,000 × 30 × 1,000 B ≈ 1.5 TB/day before compression.

How that volume turns into a bill depends on what your backend meters:

Trace sampling means keeping a subset of traces and discarding the rest. The decision has to be made once per trace and applied to every span in it. Suppose each span were sampled independently at 10%. A 30-span trace would then keep about three random spans with missing parents. That fragment shows neither the critical path nor where the latency came from. Sampling keyed on the trace ID keeps all of a trace's spans or none of them.

Before you design anything, read your own backend's pricing terms. If it bills on ingest, vendor-side sampling may lower your storage and query charges while your ingest charge stays the same. To cut ingest cost, you have to drop spans before they leave your infrastructure.

Head sampling vs tail sampling: what each can and cannot see

There are two places to drop spans before they leave your network: in the SDK, or in a collector you run.

Head sampling happens in the SDK when the root span starts. The usual setup wraps a trace-ID-ratio sampler in a parent-based sampler. The root service keeps a fixed fraction of trace IDs. Downstream services read the sampled flag from the incoming traceparent header and follow that decision, so each trace is either complete or absent. You configure it with OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1. Head sampling costs almost nothing, because unsampled spans are never exported. Its weakness is timing. It decides before anything has happened, so it cannot know whether the request will fail or run slowly.

Tail sampling runs in the collector. It buffers spans grouped by trace ID, waits for the trace to complete, and then applies policies based on error status, latency or attributes. The cost is memory and a more complex collector topology. The benefit is that the decision is based on what actually happened.

Backend sampling has the same visibility as tail sampling. Under ingest-based billing, though, you still pay for every span you send.

The two tiers interact. Tail sampling only sees traces that head sampling kept. If the SDK keeps 1%, then 99% of errors are dropped before any tail policy can evaluate them.

Copyexport OTEL_SERVICE_NAME="checkout-api"

export OTEL_TRACES_SAMPLER="parentbased_traceidratio"

export OTEL_TRACES_SAMPLER_ARG="0.1"

export OTEL_TRACES_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_ENDPOINT=""
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

Shows SDK head sampling configured with the standard OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG environment variables, keeping 10% of root traces and exporting them to a collector for optional tail sampling.

How the tail sampling processor works and how to design policies

Tail sampling inverts head sampling's order. The processor holds spans in memory, grouped by trace ID, for a fixed decision_wait period that starts when the first span of a trace arrives. When the timer expires, it evaluates every configured policy against the spans it has collected. If any policy votes to sample, the whole trace goes downstream. Otherwise all of its buffered spans are discarded. Policies combine with OR, so one broad rule can override every narrower one.

Why decision_wait governs completeness

The processor cannot tell when a trace has finished, so decision_wait has to be longer than your typical end-to-end trace duration. If it is too short, the decision runs before slow downstream spans arrive. The latency policy then measures a truncated trace, and an error raised late never gets counted. The cost is memory: every in-flight trace stays buffered for the whole wait.

Async work makes this harder. Consumers of a queue, deferred jobs and batch processors can emit spans minutes after the request that started them, long after the decision has been made. Those late spans are no longer grouped with their original trace. They are dropped or stored as orphaned fragments, so the trace you keep shows the request but not the work it triggered. Accept that gap, or link the async work as a new trace rather than parenting it to the request.

Designing the policy set

A debugging-oriented set usually contains these policies:

• Errors: a status_code policy that keeps every trace containing an ERROR span.
• Slow traces: a latency policy with a threshold set at your SLO boundary, not at the median.
• Baseline: a small probabilistic policy so you still have normal traffic to compare against.
• Rare paths: string_attribute or numeric_attribute keep rules for paths such as admin operations or key tenant IDs.
• Noisy endpoints: an and policy that pairs an endpoint match with rate_limiting, plus an exclusion that removes that endpoint from the baseline. Because policies combine with OR, the cap has no effect if the baseline still matches the endpoint.

Tuning the rules

Broad rules remove the savings you were sampling for. "Keep anything with attribute X" often matches most of your traffic. Strict rules have the opposite problem: during an incident, the degraded-but-successful traces you need may never have been kept.

Copyprocessors:
  tail_sampling:
    # Must exceed typical end-to-end trace duration, or late spans miss the decision.
    decision_wait: 30s
    # Upper bound on traces buffered in memory; size from traffic x decision_wait.
    num_traces: 100000
    expected_new_traces_per_sec: 2000
    # Policies are ORed: any single match keeps the whole trace.
    policies:
      # Always keep traces containing an ERROR span.
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      # Keep traces slower than the SLO boundary, not the median.
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 1500}
      # Small baseline of normal traffic for comparison.
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 2}
      # Rare, high-value paths: keep all admin operations.
      - name: keep-admin
        type: string_attribute
        string_attribute: {key: http.route, values: ["/admin/.*"], enabled_regex_matching: true}
      # Key tenants get full retention (placeholder IDs).
      - name: keep-key-tenants
        type: string_attribute
        string_attribute: {key: tenant.id, values: [tenant-0042, tenant-0107]}
      # Noisy endpoint: extra keeps capped by span rate. The baseline still applies
      # to it, so this rule bounds additional volume rather than dropping traffic.
      - name: cap-search
        type: and
        and:
          and_sub_policy:
            - name: is-search
              type: string_attribute
              string_attribute: {key: http.route, values: [/api/search]}
            - name: search-rate
              type: rate_limiting
              rate_limiting: {spans_per_second: 50}

A tail_sampling processor config that always keeps errors and slow traces, keeps a 2% baseline, explicitly keeps admin routes and key tenants, and rate-caps the extra traces kept from a noisy search endpoint.

Implementing the two-tier collector topology

A tail sampler can only judge traces it sees whole. If spans from one trace land on three collector replicas behind a round-robin load balancer, each replica sees a fragment. One replica keeps its fragment because it holds the error span, and the others drop theirs as unremarkable. The backend receives a broken trace that cost money and explains nothing.

The standard fix is two tiers:

apps → tier 1 (load-balancing exporter, keyed on trace ID) → tier 2 (tail sampling processor) → backend

Tier 1 is stateless. It hashes each span's trace ID and forwards the span to one tier-2 instance, so every span of a trace reaches the same sampler. In the load-balancing exporter config, set the routing key to trace ID and pick a resolver that discovers tier-2 endpoints. Use a DNS resolver pointed at a headless service, or the Kubernetes resolver, rather than a static list.

The hash ring is the weak point. When tier 2 scales, restarts or rolls out, its membership changes and some trace IDs map to a different instance. Spans that arrive after the change land on a sampler with no history for that trace. A restarted instance also loses every trace it was still buffering during its decision wait. Mitigations:

• Scale tier 2 on a schedule or with slow autoscaling, not on short spikes.
• Roll out one instance at a time, outside peak traffic.
• Keep the decision wait short to shrink the amount of in-flight data at risk.
• Accept a small rate of incomplete traces during changes, and track it.

Copyreceivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  loadbalancing:
    # Hash on trace ID so all spans of a trace land on the same tier-2 instance.
    routing_key: "traceID"
    protocol:
      otlp:
        # Settings used for each connection to a tier-2 instance.
        tls:
          insecure: true   # placeholder; use TLS in production
    resolver:
      # A headless service returns one A record per tier-2 pod.
      # When pods are added or removed, the resolved set changes and the
      # hash ring is rebuilt, so some trace IDs move to a different sampler.
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      # No tail sampling here: tier 1 forwards spans and keeps no trace state.
      exporters: [loadbalancing]

A tier-1 collector config that uses the load-balancing exporter to route spans by trace ID to tier-2 tail-sampling instances found through a headless-service DNS resolver.

Impact on cost: a worked model you can rerun

The model below uses assumed inputs. Substitute your own.

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs: 1 KB average span, $0.30/GB ingest, 30-day month. Keep rates per policy: 100% of error traces (2% of traffic), 100% of slow traces (3%), and 5% of the remaining 95%. The effective keep rate is 0.02 + 0.03 + 0.0475 ≈ 9.75%.

Formulas:

• Monthly GB = spans/s × 2,592,000 × span KB ÷ 1,000,000
• Savings = GB × price × (1 − keep rate)
• Buffered memory ≈ span rate × decision wait × average span size, plus 2× headroom for bursts and late spans

The low-volume case loses money. The fixed cost of a redundant two-tier topology does not shrink with traffic, but the savings do.

Break-even is collector cost ÷ (price × (1 − keep rate)). With these assumptions that comes to about 1,100 GB/month, or roughly 430 spans/s.

Tail sampling suits high-volume, multi-service systems on ingest-based pricing. In those systems, keeping error and slow traces in full is worth paying for the extra collector tier.

Small systems, single services, and flat-rate backends should start with head sampling only. Revisit tail sampling once volume clears the break-even point.

Who should use tail sampling?

Tail sampling fits teams that meet most of these conditions:

• They run distributed systems with real fan-out. Requests cross several services, so a complete trace is the main way to find where latency or failure originated.
• Trace volume is a meaningful line item. The backend bills on ingest or bytes, and volume is above the break-even point from your own version of the cost model.
• Rare outcomes matter more than averages. Errors, timeouts and slow requests are a small share of traffic, and fixed-ratio head sampling discards most of them.
• They can operate the collector tier. Someone owns the two-tier deployment, its memory sizing, its rollouts and its self-monitoring, in the same way they would own any other stateful production service.

It is a poor fit for teams that cannot yet run a collector at all, or whose tracing is limited to one or two services where logs and metrics already locate most problems.

When should you use tail sampling?

Use tail sampling when these conditions hold:

• Head sampling is dropping the traces you need. If incident reviews keep ending with "the failing request wasn't sampled", the decision needs to see the outcome.
• You need to cut ingest, not just storage. Backend-side sampling does not reduce an ingest bill. Tail sampling drops data before it leaves your infrastructure.
• Your keep criteria can be expressed as policies. Error status, latency above an SLO threshold, specific routes, specific tenants and a probabilistic baseline map directly onto the processor's policy types.
• Most traces complete within a bounded window. If typical end-to-end duration fits inside a decision_wait you can afford to buffer, the processor sees whole traces.
• Volume clears break-even. Rerun the cost model with your own span rate, span size, price and collector cost before committing.

When should you avoid tail sampling?

Avoid it, or defer it, in these cases:

• Low volume. As the illustrative model shows, the fixed cost of a redundant two-tier topology can exceed the ingest savings. Head sampling, or no sampling, is cheaper.
• Flat-rate or storage-only pricing. If dropping spans before export does not change the bill, the extra infrastructure buys little beyond backend-side sampling.
• Long-running or heavily asynchronous traces. If meaningful spans arrive minutes after the root, a practical decision_wait will decide on incomplete traces. Latency and error policies then misjudge them, and late spans are dropped or orphaned.
• No ability to route by trace ID. Without a load-balancing tier, multiple replicas each see fragments and produce broken traces. A single sampler instance avoids this but is a single point of failure and a memory ceiling.
• Metrics are derived from stored traces. If dashboards and alerts are computed from sampled traces, tail sampling's deliberate bias toward errors distorts them. Move RED metrics ahead of the sampler first, as described below.
• Memory is tightly constrained. The buffer scales with span rate × decision_wait × span size. If you cannot provision that plus headroom, the sampler will refuse data or be killed under load.

Combining head sampling, tail sampling and span hygiene

The three layers solve different problems, so apply them in a fixed order: hygiene first, head sampling only where volume forces it, tail sampling last.

Head sampling belongs only on services whose raw span rate would overwhelm the gateway tier, such as an edge proxy or a high-fanout internal API. Use a parent-based ratio sampler so downstream services follow the root's decision and traces stay complete. Keep the ratio conservative. Every trace dropped here is invisible to tail policies, including the errors you wanted to keep.

Span hygiene runs in the gateway pipeline, after the receiver and before tail_sampling. Place a filter processor first to drop spans that never carry diagnostic value, such as readiness probes, liveness checks and load-balancer pings, matched on route or span name. Follow it with an attributes processor to delete or hash high-cardinality keys like raw SQL, full URLs with query strings, or user identifiers. A transform processor can then truncate oversized string attributes to a fixed length.

The ordering matters because the tail sampler holds every span of every in-flight trace in memory until the decision wait expires. Spans removed upstream never occupy that buffer, which reduces the memory you size for and the num_traces headroom you tune. The saving in engineering time comes from the policy set. Without hygiene, teams write and maintain exclusion policies such as "drop traces whose only span is /healthz". With hygiene in place, those traces never arrive, so the policy list contains only outcome rules such as errors, latency and key routes. Each of those rules says something about what you want to keep.

Copyreceivers:
  otlp:
    protocols:
      grpc: {}
processors:
  # 1. Drop probe spans so they never enter the tail-sampling buffer
  filter/probes:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'
        - 'attributes["http.route"] == "/readyz"'
  # 2. Remove or hash high-cardinality keys
  attributes/cardinality:
    actions:
      - {key: db.statement, action: delete}
      - {key: url.full, action: delete}
      - {key: enduser.id, action: hash}
  # 3. Cap any remaining string attribute at 256 characters
  transform/truncate:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - truncate_all(attributes, 256)
  # 4. Outcome-only policies; no noise-exclusion rules needed
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
      - {name: slow, type: latency, latency: {threshold_ms: 500}}
      - {name: checkout, type: string_attribute, string_attribute: {key: http.route, values: [/api/checkout]}}
exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/probes, attributes/cardinality, transform/truncate, tail_sampling]
      exporters: [otlp]

A gateway collector pipeline that filters probe spans, deletes or hashes high-cardinality attributes and truncates long strings before tail_sampling, so the sampler buffers less data and needs only outcome-based policies.

Keeping metrics accurate and operating the sampler safely

Most traces a tail sampler keeps show what the system did wrong, and few of them describe typical traffic. That is the main risk of tail sampling: the retained dataset is deliberately biased toward errors and slow requests.

Derive RED metrics before sampling

Compute request rate, error rate and duration from every span, not from what survives. Place the span metrics connector in the first-tier pipeline, ahead of the tail sampling processor, so it sees the full stream. Dashboards and alerts then reflect real traffic. If they were built on sampled traces, error rates would look inflated and latency would look worse than users experience.

Weight anything computed from traces

Percentiles, endpoint popularity or "slowest customers" queries run on sampled traces will overrepresent failures. Record each trace's sampling weight, meaning the inverse of the probability under which it was kept, as an attribute. Then multiply by that weight, or use adjusted counts where your pipeline propagates them. A trace kept by an "always keep errors" rule has weight 1. A trace kept by a 5% baseline has weight 20.

Operate the sampler like a production service

Watch the collector's own telemetry: dropped or refused spans, decision latency, traces held in memory, and process memory. Put the memory limiter first in the pipeline. Under overload it then refuses data in a controlled way instead of letting the process be killed and losing every buffered trace.

Roll out policy changes to a subset of collectors first, and compare kept-trace rates per policy before and after. Review explicit keep rules whenever routes, tenants or critical flows change, because stale rules either waste budget or miss what matters.

The payoff shows up during incidents. When the kept traces are the failing and slow ones, engineers open a relevant trace immediately instead of searching through healthy requests.

How can tail sampling save engineering time?

The cost model measures money. The larger saving is often engineers' time, and it comes from a few specific mechanisms:

• Less searching during incidents. With head sampling, an engineer investigating a failure first has to find a sampled trace that shows it, and often none exists. With error and latency policies, the failing and slow traces are kept in full. Filtering by error status or duration returns relevant traces directly.
• Fewer "not sampled" dead ends. When a support ticket or alert points to a specific failed request, an error-keep policy makes it likely that the trace exists. Engineers spend less time reproducing the problem just to capture a trace.
• Complete traces instead of fragments. Trace-ID routing and a sufficient decision_wait mean a kept trace shows the whole critical path. Nobody has to stitch logs from several services together to work out where time went.
• Declarative, reviewable keep rules. The policy set lives in collector config. Teams state what they want to keep, such as errors, SLO breaches, admin routes and key tenants, in one place. They do not have to adjust sampling ratios service by service in application code.
• Smaller policy lists when hygiene runs first. Filtering probe spans and trimming attributes upstream removes the need for noise-exclusion policies. The rules that remain are fewer and each one has a clear purpose.
• Budget control without redeploying services. Raising or lowering the baseline percentage, or adjusting a rate cap, is a collector config change rolled out to the sampling tier. Instrumented services do not need to change.

These savings depend on operating the sampler well. A misconfigured decision_wait, an unbalanced hash ring or metrics derived from sampled traces can cost more debugging time than they save. Treat the sampling tier as production infrastructure from the start.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to opentelemetry tail sampling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.