Why can't head sampling reliably keep errors and slow requests?

Head sampling makes its keep-or-drop decision when the root span starts. The SDK, or a sampler at the edge, rolls the dice and propagates the result through the trace context. At that moment nothing is known about how the request will end: whether a downstream call will time out, whether the database will return an error, or whether the response will take 40 ms or 4 seconds. The decision is therefore independent of outcome. A 10% head sample keeps roughly 10% of your failures and roughly 10% of your slowest requests, which are the traces you most need during an incident. Rare errors may not be captured at all.

Tail sampling moves the decision to the end. The OpenTelemetry Collector buffers every span for a trace, waits until the trace is assembled, and then evaluates policies against the complete picture: status codes, total duration, attributes, span counts. You can keep every trace containing an error and every trace over your latency threshold, and sample the healthy majority at a low rate.

The cost is that tail sampling is a stateful, memory-bound buffering system, not a config toggle. The Collector has to hold spans in memory while it waits, and every span of a trace must reach the same instance. The rest of this article covers how to handle both constraints.

Who should use tail sampling?

Tail sampling suits backend and platform engineers running high-volume services, where storage cost forces sampling but errors and SLO breaches must be retained. During an incident, the failing traces are already stored, so engineers can debug from them instead of trying to reproduce the problem.

It is a good fit when all of the following are true:

  • Trace volume is high enough that you must sample. If storing every span is affordable, there is nothing for a sampler to save.
  • Failures and slow requests are a small fraction of traffic. This is where head sampling loses the most: the rarer the error, the more likely a fixed-rate head sample misses it.
  • You have latency SLOs to anchor policies to. Latency policies work best when their thresholds come from an objective someone owns.
  • Most traces complete within seconds to a few tens of seconds. Request/response services fit the buffering model. Long queue-driven workflows do not (see the last section).
  • A team can own the Collector tier. Someone has to run the routing and sampling tiers, size their memory, and review policies as services change.

In practice that usually means SRE and platform teams operating shared Collector infrastructure for many services, and on-call engineers who need complete traces for every error.

How does tail sampling work in the tail_sampling processor?

Deciding after the fact means the Collector has to hold spans until it has enough of a trace to judge it. The tail_sampling processor does this in four steps:

  1. Buffer by trace ID. Incoming spans are grouped in memory by trace ID, and each trace gets its own buffer.
  2. Wait decision_wait. A timer starts when the first span of a trace arrives. Later spans are appended to the buffer until the timer expires. The timer does not detect that a trace is complete. It assumes that most spans of a trace arrive within that window.
  3. Evaluate policies. When the timer expires, every configured policy runs against the assembled trace, including status-code checks, latency checks and probabilistic rules.
  4. Forward or drop. The whole trace is either exported or discarded. Individual spans are never sampled separately.

Spans that arrive after the decision are handled using the remembered decision, if the processor still has one. If it does not, a late span is treated as a new trace and evaluated with incomplete data. A slow downstream batch can therefore be judged without its error-carrying parent.

Memory is bounded by num_traces. When that limit is reached, the oldest traces are evicted before any policy runs, and their spans are lost.

None of this works unless every span of a trace reaches the same Collector instance. If spans are split across instances, each one sees a fragment and makes its own decision. The target design is therefore two tiers. A stateless first tier receives telemetry and uses a trace-ID-aware load-balancing exporter to route spans. A second tier runs tail_sampling, and each of its instances owns a consistent share of trace IDs.

  +---------+        +---------+        +---------+
  | svc A   |        | svc B   |        | svc C   |
  +----+----+        +----+----+        +----+----+
       |                  |                  |
       v                  v                  v
  +------------------------------------------------+
  | TIER 1: stateless Collectors (any count)       |
  | otlp receiver -> loadbalancing exporter        |
  | routing key: trace ID (consistent hashing)     |
  +----------+-------------------------+-----------+
             |                         |
             | trace IDs share A       | trace IDs share B
             v                         v
  +----------+-----------+  +----------+-----------+
  | TIER 2: Collector #1 |  | TIER 2: Collector #2 |
  | tail_sampling        |  | tail_sampling        |
  | buffer per trace ID  |  | buffer per trace ID  |
  | wait decision_wait   |  | wait decision_wait   |
  | evaluate policies    |  | evaluate policies    |
  | export/drop trace    |  | export/drop trace    |
  | cap: num_traces      |  | cap: num_traces      |
  +----------+-----------+  +----------+-----------+
             |                         |
             v    sampled traces only  v
  +------------------------------------------------+
  |                tracing backend                 |
  +------------------------------------------------+
  late span: reuse remembered decision, else new trace
  num_traces full: oldest trace evicted, spans lost

Stateless tier-1 Collectors route spans by trace ID so that each tier-2 Collector receives complete traces, buffers them for decision_wait, evaluates its policies, and exports or drops each trace as a whole.

How do you write tail sampling policies for errors and latency?

Start the pipeline with memory_limiter before tail_sampling. The tail sampler holds whole traces in memory until it decides, so when the Collector runs short on memory, the limiter needs to refuse incoming data first. Otherwise the process gets killed and every buffered trace is lost.

Inside tail_sampling, define three policies:

  • Errors: a status_code policy that matches ERROR. It keeps any trace where at least one span recorded a failure, including failures in a downstream call that the root span never surfaced.
  • Latency: a latency policy whose threshold comes from your SLO. If checkout promises 800 ms at p99, set the threshold at or just below that so you keep traces that breach or nearly breach the objective. An arbitrary round number has no link to user impact.
  • Baseline: a probabilistic policy at a low percentage. It keeps a sample of healthy traffic so you can compare good requests with bad ones.

Each policy is evaluated independently, and a trace is sampled if any policy votes to keep it. A slow trace that also errored is kept once, not twice. Composite and and-style policies work differently. An and policy requires all of its sub-policies to match. A composite policy allocates a share of throughput to each sub-policy, which caps how much any single rule can consume.

Policy types and options change between releases, so check the tail_sampling README for your Collector version before copying a config.

Plan for maintenance. SLO targets change, latency profiles drift after deploys, and per-service thresholds tend to multiply. Keep the thresholds in version control next to the SLO definitions they come from.

receivers:
  otlp:
    protocols:
      grpc:

processors:
  # Runs first: refuses data under memory pressure instead of letting the process be killed
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300
  tail_sampling:
    decision_wait: 10s          # how long to buffer spans before deciding on a trace
    num_traces: 50000           # max traces held in memory at once
    policies:
      # Keep any trace where at least one span has status ERROR
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      # Threshold mirrors the checkout SLO (p99 800 ms); update both together
      - name: keep-slo-breaches
        type: latency
        latency:
          threshold_ms: 750
      # Low-rate baseline of healthy traffic for comparison
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch:

exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

A Collector traces pipeline with memory_limiter ahead of tail_sampling. The pipeline keeps every errored trace, every trace near or over the 800 ms checkout SLO, and a 5% baseline of healthy traffic; check the field names against the tail_sampling README for your Collector version.

How should you choose decision_wait and size Collector memory?

Start from measured data, not a round number. Take the p99 (or p99.9) end-to-end trace duration from your current backend, then add the worst-case delay before the last span reaches the Collector: SDK batch export interval, agent hops, and retries. A trace that runs 8 s, with spans that can arrive 5 s late, needs a decision_wait of at least 13 s. Otherwise the policy judges an incomplete trace.

decision_wait Effect
Too short Slow and async traces get evaluated before their slow spans arrive, so latency policies miss them. Late spans end up orphaned or in split traces.
Too long More traces held in memory. Sampled data appears in the backend later, which slows incident response.

No single value fits services that differ widely in trace duration. A 200 ms API and a 10-minute batch job sharing one Collector force a compromise. Route those workloads to separate pipelines or Collector pools instead.

Illustrative estimate (assumed inputs, not measured):

  • Assume 5,000 spans/s, averaging 10 spans per trace. That is 500 new traces/s.
  • With a 30 s decision_wait, about 15,000 traces and 150,000 spans are in flight.
  • At an assumed 1 KB per span in memory, that is about 150 MB.
  • Double it for bursts: set num_traces around 30,000 and budget about 300 MB plus Collector baseline.
  • Set memory_limiter limits above that figure, not below it.

Hitting either limit loses data without an obvious failure. When num_traces is exceeded, the processor evicts the oldest traces before they are evaluated, and errors can vanish with them. When memory_limiter triggers, it refuses incoming data, and upstream senders may drop spans. Alert on the Collector's dropped-trace and refused-span metrics.

receivers:
  otlp:
    protocols:
      grpc:

processors:
  # Runs first: refuses data before the heap runs out. Upstream senders may drop spans.
  memory_limiter:
    check_interval: 1s
    limit_mib: 512        # Above the ~300 MB in-flight estimate plus Collector baseline
    spike_limit_mib: 128  # Soft limit = 512 - 128 = 384 MiB

  tail_sampling:
    # p99 trace duration + worst-case late arrival, rounded up (illustrative: 30s)
    decision_wait: 30s
    # ~500 traces/s x 30s = 15,000 in flight; doubled for bursts.
    # Beyond this, the oldest traces are evicted before they are evaluated.
    num_traces: 30000
    expected_new_traces_per_sec: 500
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

  batch:

exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp]

A Collector traces pipeline whose decision_wait, num_traces and memory_limiter values come from the illustrative estimate (500 traces/s, 30 s wait, ~1 KB per span, 2x headroom), with memory_limiter placed first so it can refuse data before the process runs out of memory.

How do you scale tail sampling horizontally without breaking traces?

A single Collector eventually runs out of CPU or memory, but adding replicas behind an ordinary round-robin load balancer breaks tail sampling without any error. The tail_sampling processor can only evaluate the spans it has received. If one trace's spans land on three instances, each instance sees a fragment. The instance holding the failing span may keep its piece. The other two may drop theirs, or keep them under a probabilistic policy, and the backend receives an incomplete trace. A latency policy fails the same way: no single instance sees both the root span's start and the slowest child, so a slow request can look fast everywhere. Nothing is logged because every instance is behaving correctly with the data it has.

The two-tier layout

The fix is to separate routing from sampling. The front tier is a stateless set of Collectors that runs the load-balancing exporter with routing by trace ID. The exporter hashes each span's trace ID onto a consistent-hash ring of back-tier endpoints, so every span of a trace reaches the same instance. The back tier runs tail_sampling and holds the buffered traces. Because the front tier keeps no buffer, you can scale it freely.

The resolver tells the exporter which back-tier endpoints exist:

  • Static list: fixed addresses that are simple to predict, but you must edit the list and redeploy whenever the back tier changes.
  • DNS or Kubernetes service discovery: the exporter tracks the back-tier endpoints automatically. In Kubernetes, point it at a headless service so it sees individual pod IPs rather than a single virtual IP.

Resharding risk

Any change to back-tier membership rehashes part of the trace-ID space. This includes scale-up, scale-down, and pod restarts. Traces in flight during the change get split between the old owner and the new one, and both decide on a partial trace. A pod that terminates also loses whatever it was still buffering. To limit the damage:

  • Scale the back tier during low-traffic periods and in small steps.
  • Set a termination grace period longer than decision_wait so a terminating pod can finish evaluating what it has buffered.
  • Use disruption budgets so node maintenance doesn't restart many back-tier pods at once.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  loadbalancing:
    # Hash on trace ID so every span of a trace lands on the same back-tier pod.
    routing_key: "traceID"
    protocol:
      otlp:
        tls:
          insecure: true   # placeholder; use real TLS settings in production
    resolver:
      # DNS against a *headless* service returns individual pod IPs,
      # so membership changes are picked up without a redeploy.
      dns:
        hostname: otelcol-sampler-headless.observability.svc.cluster.local
        port: 4317
      # Alternative: a fixed list. Predictable, but you must edit and
      # redeploy whenever the back tier changes.
      # static:
      #   hostnames:
      #     - sampler-0.example.internal:4317
      #     - sampler-1.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]

A front-tier Collector config that uses the load-balancing exporter to route spans by trace ID to back-tier tail-sampling instances, discovered through DNS on a headless service, with a static list shown as the alternative.

How do you verify and roll out tail sampling safely?

What to watch in the Collector's own telemetry

The tail_sampling processor reports its own metrics through the Collector's internal telemetry. Exact metric names change between releases, so check what your build exposes before you build dashboards. Watch four signals:

  • Sampled vs. dropped decisions per policy. These show whether each policy is matching anything at all. A latency policy that never fires usually means the threshold or the span it measures is wrong.
  • Late spans. These are spans that arrive after their trace was already decided. A steady rate means decision_wait is too short, or load balancing is splitting traces across instances.
  • Traces held in memory. This should plateau. If it keeps climbing, throughput has outgrown the buffer.
  • memory_limiter refusals. Any refusal means data was rejected before sampling ran. Errors can be lost here without the sampler ever seeing them.

Prove it with a known error and a known slow request

Add a test endpoint that returns a 500, and another that sleeps well past your latency threshold. Call each one with a recognizable attribute or trace ID. Then confirm that both traces arrive in the backend complete, with every service's spans present. Run the test again after every policy change.

Roll out permissively first

Begin with a broad policy set. Run it alongside full export, either to a cheap sink or for a short window. Then compare the error and slow traces the sampler kept against the unsampled copy. Tighten the probabilistic baseline only after the two match.

Compute rates before sampling

Sampled data overrepresents errors and slow requests, which skews error rates and latency percentiles. Derive request rate, error rate, and latency percentiles from metrics produced upstream of the sampler. You can generate them in the SDKs or with a connector placed ahead of tail_sampling in the pipeline.

How can tail sampling save engineering time?

The time savings come from having the right traces already stored when someone needs them, rather than from the sampler itself.

  • No reproduction step for errors. With head sampling, a failing request has a good chance of being dropped, so an engineer often has to reproduce the failure, temporarily raise the sampling rate, or add logging and redeploy. With an error policy, every errored trace is retained, and debugging starts from the actual failing request.
  • Cross-service failures arrive as one trace. Because the decision covers the whole trace, a timeout three services deep is stored together with the root span and every hop in between. Engineers don't have to correlate fragments across services by timestamp.
  • Slow requests are kept by definition. A latency policy tied to the SLO means traces behind a latency alert are available immediately. Nobody has to wait for a slow request to recur while sampling happens to be favorable.
  • Fewer sampling-rate changes during incidents. Teams relying on head sampling often raise rates during an incident and lower them afterwards, which means deploys or config changes across many services. Tail policies live centrally in the Collector, so one config change covers every service that sends to it.
  • Less noise to search through. Healthy traffic is reduced to a small baseline, so queries for errors or slow requests return a higher proportion of useful traces.

These savings depend on the setup being correct. Split traces, evicted traces and memory_limiter refusals all remove exactly the traces engineers need, which is why the verification steps in the previous section matter. Also budget for the time the system itself takes: sizing, operating the routing tier, and reviewing policies as SLOs change.

When should you avoid tail sampling, and what are the alternatives?

Tail sampling is not always worth it. Skip it in these cases:

  • Low-volume services. Storing every span can cost less than running and tuning a sampling tier.
  • No operational capacity. Sampling depends on a load-balancing tier routing all spans of a trace to one Collector. If nobody owns that tier, it becomes a single point of failure.
  • Long async or queue-driven traces. When consumers emit spans minutes after the producer, no decision_wait covers the gap. Late spans miss the decision and arrive as orphaned fragments.
  • SDK and network overhead is the real cost. Every span is still created, serialized and exported to the Collector before any decision is made. Tail sampling cuts backend ingest and storage, not instrumentation or egress.
Approach Error and latency coverage Main cost
Head sampling Probabilistic; rare errors often lost Lowest; reduces SDK and network load
Head plus tail Only among head-sampled traces Two layers of configuration
No sampling, shorter retention Complete, for a shorter window Ingest volume unchanged
Metrics and exemplars Aggregates complete; traces linked by example only Requires metrics pipeline and exemplar support
Backend-side sampling Depends on backend Full ingest and transfer still billed

Tail sampling adds ongoing costs. Collectors need more memory and CPU to buffer traces. The topology gains a routing layer. Policies need regular review as services and SLOs change. In return, you get lower backend ingest and storage, and engineers spend less time searching for error traces that head sampling dropped. If that search rarely happens on your team, one of the alternatives above is likely the better choice.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to opentelemetry tail sampling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.