What is OpenTelemetry tail sampling and why do backend APIs need it?

Sampling decides which traces your tracing pipeline keeps. With head sampling, that decision happens when the root span starts, usually by hashing the trace ID against a fixed ratio, and the choice propagates to downstream services through trace context. It is cheap and stateless. The cost is that the decision happens before anything about the request is known. At a 5% ratio, a request that returns a 500 after timing out on a database call has the same 5% chance of being kept as a fast 200. The traces you most want for debugging are the rarest, so they are the most likely to be dropped.

OpenTelemetry tail sampling moves the decision to the end. The otel collector buffers every span sharing a trace ID, waits for the trace to finish, then evaluates policies against the assembled trace: error status, end-to-end latency, HTTP route, customer tier, or any other attribute. A few rules cover most needs: keep every error, keep anything over the latency threshold, and keep a small baseline of healthy traffic.

Aspect Head sampling Tail sampling
Decision point Root span start After trace completes
Information available Trace ID, initial attributes Status, duration, all span attributes
Error/outlier retention Same rate as healthy traffic Can keep all of them
State required None Buffered spans per trace
Deployment constraint Any topology All spans of a trace must reach one collector

This matters most for teams running moderate-to-high traffic APIs, where storing every trace is too expensive but missing a rare failure is unacceptable. The policies are easy to write. The difficult engineering is in the last row of the table: routing every span of a trace to the same collector instance and sizing the buffer that holds them.

How does the tail sampling processor work inside the OTel Collector?

The processor does not decide anything when a span arrives. It holds spans in memory, grouped by trace ID, and starts a timer when it sees the first span of a new trace. When the configured decision_wait expires, it evaluates every policy against all spans collected for that trace. That is how a policy can say "keep this trace if any span has an error status" or "keep it if the root span took longer than 2 seconds." The decision applies to the trace as a unit, and the buffered spans are then forwarded or dropped together.

This design assumes a trace finishes within the wait window, and backend systems often break that assumption. A queue consumer may process a message minutes after the producer's span ended. An async job may continue a trace long after the HTTP request returned. SDK batch exporters also add their own flush delay before a span leaves the process. Spans that arrive after the decision are late spans. Depending on configuration, the processor either applies a remembered decision to them or treats them as the start of a new, fragmentary trace. In the second case, policies evaluate an incomplete trace, and you get orphan spans in your backend or lose the interesting part of the trace.

Because the processor holds per-trace state, two constraints follow:

  • Every span of a trace must reach the same instance. If spans are split across replicas, each replica judges a partial trace, and a replica that never sees the error span drops its half of the trace.
  • The state is not durable by default. Undecided traces exist only in process memory. A crash, restart, or OOM kill discards every trace still waiting for a decision.

Memory pressure is also a risk. When the buffer reaches its trace limit, the oldest traces are evicted. Sizing the buffer is therefore part of making the pipeline correct, not only part of tuning it.

How do you build a two-tier tracing pipeline with trace-ID load balancing?

The fix for split traces is a pipeline with two tiers. The first tier receives OTLP from your services and holds no trace state. Its only job is routing. It uses the load-balancing exporter with routing_key set to traceID. The exporter hashes each span's trace ID onto a consistent-hash ring of second-tier endpoints, so every span of a trace reaches the same backend. The second tier runs the tail sampling processor. Because each instance there sees complete traces, its decisions are valid. First-tier instances can sit behind any ordinary load balancer and scale freely, since they buffer nothing.

The exporter needs to know which second-tier instances exist:

Resolver How it discovers backends Strengths Weaknesses
Static (hostnames list) Fixed list in config Predictable, easy to reason about Changing membership requires a config change and restart
DNS (e.g. Kubernetes headless service) Periodically resolves a hostname to all pod IPs Follows pod churn automatically Refresh interval causes brief disagreement between first-tier instances

Any membership change remaps part of the hash ring. During scale-up, scale-down or a rolling restart, spans that arrive after the change can go to a different backend than the spans that arrived before it. In-flight traces then split, and each fragment receives its own sampling decision. A terminated pod also discards whatever it was buffering.

This is why autoscaling the sampling tier is harder than it looks. CPU-driven autoscalers react to load spikes by adding pods, and every resize splits traces at exactly the moment when traffic is most interesting. A more reliable approach is to keep the sampling tier deliberately overprovisioned, scale it rarely, and drain instances for at least the decision wait period before removing them.

# --- Tier 1: stateless router (otel-gateway) ---
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
exporters:
  loadbalancing:
    routing_key: traceID          # all spans of a trace hash to one backend
    protocol:
      otlp:
        tls: { insecure: true }   # placeholder; use TLS in production
    resolver:
      # Static alternative: fixed ring, changes need a config edit and restart
      # static: { hostnames: [sampler-0:4317, sampler-1:4317] }
      dns:                        # headless service returns every pod IP
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317
        interval: 5s              # refresh period; instances may briefly disagree
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [loadbalancing] }
---
# --- Tier 2: stateful sampler (otel-sampler), sees complete traces ---
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
processors:
  tail_sampling:
    decision_wait: 10s            # buffer spans this long before deciding
    num_traces: 100000            # in-memory trace capacity per instance
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow, type: latency, latency: { threshold_ms: 500 } }
      - { name: baseline, type: probabilistic, probabilistic: { sampling_percentage: 10 } }
exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317
service:
  pipelines:
    traces: { receivers: [otlp], processors: [tail_sampling], exporters: [otlp] }

Two collector configs show the pipeline: a stateless first tier that routes spans by trace ID with the load-balancing exporter (DNS resolver active, static resolver in a comment), and a second tier that runs tail sampling on complete traces.

How should you write tail sampling policies for API traffic?

Start with four top-level policies, each answering a different question about a trace:

  • Errors: a status_code policy matching ERROR. Make sure your instrumentation sets error status on server spans for 5xx responses. Client errors (4xx) usually should not count as errors, or every bad request will be retained.
  • Latency: a latency policy with a threshold tied to your SLO, not an arbitrary round number. If the p99 objective is 800 ms, keeping everything above 800 ms captures the traces that actually burn error budget.
  • Targeted attributes: a string_attribute policy on http.route for critical endpoints such as checkout or auth, or on a tenant ID you are actively debugging. Remove tenant rules when the investigation ends.
  • Baseline: a probabilistic policy at a low percentage. This keeps a representative sample of healthy traffic, so you can still see what normal looks like.

Top-level policies are OR'd. If any policy returns a sample decision, the trace is kept. Because of this, adding a policy can only increase volume.

Broad error and latency policies keep the most traces during an incident, which is exactly when your backend is under the most ingestion pressure. A composite policy addresses this. It sets a total spans-per-second budget and allocates a share to each sub-policy. Ordering the sub-policies controls which ones get budget first. You can also wrap individual branches in rate_limiting.

Caps have a cost, though. When the limit is hit, the processor drops traces it would otherwise keep, including error traces you wanted. Size the error share generously.

Every policy is evaluated against every buffered trace. Regex matching and long lists of attribute values add measurable CPU to the sampling tier. Prefer exact matches on low-cardinality attributes.

processors:
  tail_sampling:
    decision_wait: 10s          # time to buffer spans before deciding
    num_traces: 100000          # traces held in memory on this sampling tier
    policies:
      # One composite policy so every branch shares a single spans/sec budget.
      # Separate top-level policies would be OR'd and bypass the cap.
      - name: api-traffic
        type: composite
        composite:
          max_total_spans_per_second: 5000
          policy_order: [errors, slow, critical-routes, baseline]
          composite_sub_policy:
            - name: errors                # server spans set ERROR on 5xx only
              type: status_code
              status_code: { status_codes: [ERROR] }
            - name: slow                  # matches the 800 ms p99 SLO
              type: latency
              latency: { threshold_ms: 800 }
            - name: critical-routes       # exact matches, no regex, to limit CPU
              type: string_attribute
              string_attribute:
                key: http.route
                values: [/api/checkout, /api/auth/login]
            - name: baseline              # small sample of healthy traffic
              type: probabilistic
              probabilistic: { sampling_percentage: 2 }
          rate_allocation:              # errors get the largest share, since
            - policy: errors              # traces over budget are dropped
              percent: 50
            - policy: slow
              percent: 25
            - policy: critical-routes
              percent: 15
            - policy: baseline
              percent: 10

A tail_sampling processor config that wraps error, latency, critical-route and probabilistic policies in one composite policy, so they share a spans-per-second budget and error traces get the largest share.

Processor ordering and buffer sizing for the sampling tier

Order the sampling tier's processors as: memory_limiter, then attribute or resource processors, then tail_sampling, then batch.

Why this order:

  • memory_limiter goes first. It can refuse data before the sampler buffers it. Placed anywhere else, spans are already held in memory when the limit trips.
  • Attribute and resource processors go before the sampler. If a policy matches on a normalized route, a tenant tier, or deployment.environment, those values must exist on the span when the policy runs. A processor that adds them after the sampler has no effect on the decision.
  • Don't batch before tail_sampling. The sampler regroups spans by trace ID, so a batch processor in front of it adds nothing. It only adds delay, and that delay counts against decision_wait. Child spans held in a batch can arrive after their trace has already been decided.
  • batch goes after the sampler. At that point it compresses only the traces you kept.

Sizing the buffer (illustrative estimate)

All inputs below are assumptions:

Input Assumed value
Spans per second per instance 5,000
decision_wait 30 s
Average in-memory span size 1.5 KB

5,000 × 30 × 1.5 KB ≈ 225 MB of buffered spans. Add headroom for decision bookkeeping and traffic bursts, and set memory_limiter well above that figure.

num_traces caps how many traces are held, regardless of memory. Assuming 10 spans per trace, 500 traces/s × 30 s ≈ 15,000 traces in flight. If num_traces sits below that, the oldest traces are evicted before they are decided. Set it to two or three times the estimate.

Choosing decision_wait

  • Longer waits catch slow requests, but they cost memory and delay data reaching the backend.
  • Shorter waits split long-running or async traces. Late spans get a separate decision, which can leave traces incomplete.
processors:
  memory_limiter:            # first: refuse data before the sampler buffers it
    check_interval: 1s
    limit_mib: 1000          # well above the ~225 MB illustrative buffer estimate
    spike_limit_mib: 200
  resource:                  # add attributes the policies match on
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
  tail_sampling:
    decision_wait: 30s       # value used in the sizing estimate
    num_traces: 40000        # ~2-3x the ~15,000 traces assumed in flight
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: keep-prod-sample
        type: string_attribute
        string_attribute: {key: deployment.environment, values: [production]}
  batch:                     # last: batches only the traces that were kept
    send_batch_size: 8192
    timeout: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      # Order matters: no batch ahead of tail_sampling
      processors: [memory_limiter, resource, tail_sampling, batch]
      exporters: [otlp]

Sampling-tier Collector config with memory_limiter first, resource enrichment before tail_sampling, and batch after the decision. The decision_wait, num_traces and memory limit values follow the illustrative sizing estimate. Receiver and exporter definitions are omitted.

How do you keep RED metrics and SLOs accurate after trace sampling?

Tail sampling is biased on purpose. Your policies keep every error and every slow request but only a fraction of healthy ones, so the stored traces describe the tail, not normal traffic. A dashboard that computes error rate or p99 latency by counting stored traces will report numbers far worse than users experience. Someone will then spend an afternoon investigating an incident that never happened.

Derive RED metrics before sampling

Compute rate, errors and duration from the full span stream in the first tier, before any data is dropped. The span metrics connector does this. Add it as an exporter on the first-tier traces pipeline and as a receiver on a metrics pipeline. Every span then updates request counts, error counts and latency histograms keyed by service, span name and status code. The same spans still flow to the load-balancing exporter. Each first-tier instance sees only part of the traffic, so sum the series across instances in the metrics backend.

Keep the dimensions coarse. Use route templates rather than raw URLs. High-cardinality attributes turn this into a memory problem.

Point SLOs and alerts at these metrics. Use traces to explain what the metrics show, not to measure it.

Know your effective sample rates

Write down the keep rate for each policy, and watch the processor's per-policy decision counts. When someone sees 40 stored traces for an endpoint, they need to know whether that means 40 requests or roughly 4,000.

Head sampling upstream

If SDKs head-sample at 10%, the collector only ever receives 10% of traces. Errors in the other 90% are gone, and no tail policy can recover them. Your span metrics undercount by the same factor. Configure SDKs to send everything, or generate RED metrics in the application itself.

# First-tier collector: derive RED metrics from every span, then forward
# spans to the tail-sampling tier. Nothing is dropped in this tier.
# SDKs should send everything (e.g. OTEL_TRACES_SAMPLER=always_on).
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
    # service.name, span.name, span.kind and status.code are included by default.
    # Add only low-cardinality attributes: a route template, never a raw URL.
    dimensions:
      - name: http.route

exporters:
  # Route all spans of a trace to the same second-tier instance.
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-tier2-headless.observability.svc.cluster.local
  # Pre-sampling RED metrics; sum series across tier-1 instances downstream.
  otlp/metrics:
    endpoint: metrics-backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics, loadbalancing]  # same spans feed both
    metrics:
      receivers: [spanmetrics]
      exporters: [otlp/metrics]

A first-tier OpenTelemetry Collector config that computes RED metrics from all spans with the span metrics connector before forwarding the same spans to the tail-sampling tier via the load-balancing exporter.

How do you operate tail sampling in production and verify it works?

Tail sampling fails quietly. A sampler that evicts traces too early or an exporter that drops batches still produces traces in the backend, just fewer and less complete ones. The collector's own telemetry is the only way to tell the difference.

What to watch on the sampling tier

  • Dropped and refused spans. Rising refusals at receivers mean the memory limiter is shedding load. Spans that arrive after their trace's decision has been made, or traces evicted before the decision wait elapsed, mean decision_wait or num_traces is too small.
  • Sampling decisions per policy. Watch the sampled and not-sampled counts for each policy. If the error policy's sampled count drops to zero while your API's error rate is non-zero, something upstream is broken.
  • Memory usage. Compare it against the limiter's thresholds, not only against the container limit.
  • Exporter queue and retries. A queue that stays near capacity, or send failures that keep climbing, means the backend cannot keep up. Data loss will follow.

Set memory_limiter below the container's memory limit, with enough headroom for the Go runtime and in-flight batches. If you skip this, the kernel OOM-kills the collector and every buffered trace is lost at once. With it, the collector refuses new data, and senders can retry.

Verify end to end

Send a deliberately failing request, such as a route that returns a 500 from a downstream call. Then confirm the trace reaches the backend with every span from every service. A complete trace shows that load balancing, the decision wait, and the error policy all work together.

Run this check after every config change. During an incident, you then already know what the sampler keeps and drops, so you can trust the traces in front of you instead of first investigating the pipeline.

When should you avoid tail sampling, and what are the alternatives?

Tail sampling costs real engineering effort. You run a second, stateful collector tier. You keep the load-balancing exporter's resolver in sync with service discovery. You size memory for in-flight traces, and you monitor components that can drop spans without any visible error. Head sampling, by contrast, is one ratio in the SDK and adds nothing new to operate.

Avoid tail sampling when:

  • Traffic is low. If keeping 100% of traces costs less than the engineer-hours spent maintaining the sampling tier, keep everything.
  • No one owns the collector fleet on call. A stateful tier that loses spans during a bad rollout is worse than a predictable head ratio.
  • Your questions are mostly aggregate, such as "is p99 latency rising?" Metrics answer those better than any trace sample.
Approach Error capture Cost Complexity Completeness
Head sampling Random, so rare errors are often missed Low Minimal Whole traces, but only a fraction of them
Keep all traces Every error Scales linearly with traffic Minimal Complete
Backend-side sampling Good, because the backend sees whole traces You still pay to ship full volume Low for you Complete until the backend drops data
Metrics-first with exemplars Errors counted exactly, with linked sample traces Low Moderate Aggregates exact, traces sparse
Tail sampling Every error your policies match Moderate High Whole traces, chosen by policy

Tail sampling is worth the extra engineering only when three conditions all hold:

  1. Trace volume makes keeping everything expensive.
  2. Errors and slow requests are rare enough that a head ratio would miss them.
  3. A team has capacity to operate the sampling tier.

If any of these is false, start with head sampling plus metrics with exemplars, and revisit the decision as traffic grows.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to opentelemetry tail sampling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.