What is tail sampling, and why does head sampling miss the traces you need?

Head sampling makes the keep-or-drop decision when the root span starts, usually with a ratio sampler that hashes the trace ID. Downstream services honour that choice through the sampled flag in the propagated trace context. This approach is cheap and consistent across services. Its limitation is that the decision happens before the request has done anything. A request that will later time out, return a 500, or hit a slow database query is kept at exactly the same rate as a healthy one. At a 1% ratio, you keep roughly 1% of your rare failures too.

Tail sampling moves the decision to the end of the trace. A collector buffers all spans for a trace ID, waits for the trace to finish, then evaluates the complete trace. It can keep every trace with an error status, every trace above a latency threshold, or every trace touching a critical route, and sample the rest at a low ratio.

Aspect Head sampling Tail sampling
Decision point Root span creation After trace completes
Inputs available Trace ID, initial attributes Status, latency, route, all spans
Rare errors kept At base ratio Reliably, by policy
Infrastructure cost Negligible Buffering memory, routing tier

This is the core tradeoff behind observability costs. With head sampling alone, you either pay to store mostly uninteresting traces or lose the ones you need for debugging.

Tail sampling pays off most for high-volume services with meaningful error and latency outliers. For low-volume systems where retaining every trace is already cheap, the extra collector infrastructure isn't worth it.

How should you define trace retention tiers by purpose?

A single retention period forces a bad trade: keep everything long enough to be useful in a postmortem and you pay for weeks of healthy, redundant requests; keep it short and the trace you need is gone. Define tiers by what each one is for instead.

Tier Contents Purpose Retention (example)
Baseline Small probabilistic sample of normal traffic "What does healthy look like?" comparisons Days
Diagnostic Errors, latency outliers, flagged tenants or routes Incident debugging and postmortems Weeks
RED metrics Rate, errors, duration derived from 100% of spans Dashboards, alerts, SLOs Long-term, at metric cost

The periods shown are placeholders. Set them from how far back your team actually investigates.

The metrics path is the one teams most often get wrong. Generate RED metrics from spans before any sampling decision, for example with the span metrics connector in the first collector layer. Once tail sampling runs, the surviving traces are deliberately skewed toward failures and slow requests. Computing request rate or error ratio from them inflates error rates and understates throughput, and there is no reliable way to reweight the result. When metrics come from unsampled spans, you can change sampling policies freely without breaking dashboards or SLO burn calculations.

The tiers also save time during incidents. An alert fires on an error-rate metric, and the matching error traces are already in the diagnostic tier, filterable by status or latency. Engineers don't have to page through sampled noise or hope a slow request happened to be kept. The baseline tier then provides healthy traces of the same route to compare against.

How does the OTel Collector tail sampling processor work?

The processor holds every incoming span in memory, grouped by trace ID, until a decision wait period has elapsed since the first span of that trace arrived. Only then does it evaluate the complete set of spans against your policies. A policy can match on an error status code, a latency threshold measured across the trace, an attribute value such as tenant or route, a probabilistic percentage, or a rate limit. An and policy requires several conditions to hold at once, for example "checkout route AND latency above threshold." A composite policy divides throughput between sub-policies. If any top-level policy votes to sample, the processor keeps the trace.

The tiers from the previous section map directly onto policies:

  • Error tier: a status-code policy on errors.
  • Slow tier: a latency policy with the tier's threshold.
  • Critical-path tier: an and policy combining a route attribute with a lower latency bar.
  • Baseline tier: a small probabilistic policy, with a rate limit as a ceiling during traffic spikes.

Policy type names and field names have changed between releases, so check them against the documentation for the Collector version you run.

The wait period sets a trade-off. Queue consumers, batch jobs, and fire-and-forget async work often emit spans long after the request that started them, and clock skew between hosts makes timing less predictable. Spans that arrive after the decision may be forwarded using a cached decision or treated as a new partial trace, depending on configuration and version. A longer wait captures more complete traces but holds more spans in memory and delays when traces reach storage. Choose the wait based on your typical request duration, not on your slowest batch job.

# Field names below must be checked against the docs for your Collector version.
processors:
  tail_sampling:
    decision_wait: 10s        # buffer per trace ID; size to typical request duration, not batch jobs
    num_traces: 100000        # max traces held in memory while waiting for a decision
    expected_new_traces_per_sec: 1000  # helps pre-size internal buffers
    policies:                 # top-level policies are OR'd: any sample vote keeps the trace
      # Error tier: keep every trace with an error status
      - name: error-tier
        type: status_code
        status_code: {status_codes: [ERROR]}
      # Slow tier: keep traces whose duration exceeds the slow threshold
      - name: slow-tier
        type: latency
        latency: {threshold_ms: 2000}
      # Critical-path tier: checkout route AND a lower latency bar
      - name: critical-path-tier
        type: and
        and:
          and_sub_policy:
            - name: checkout-route
              type: string_attribute
              string_attribute: {key: http.route, values: [/api/checkout]}
            - name: checkout-slow
              type: latency
              latency: {threshold_ms: 500}
      # Baseline tier: small random sample of everything else.
      # A separate top-level rate_limiting policy would add traces, not cap them;
      # to enforce a ceiling during spikes, wrap tiers in a composite policy.
      - name: baseline-tier
        type: probabilistic
        probabilistic: {sampling_percentage: 2}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp]

An annotated tail_sampling processor configuration that maps the error, slow, critical-path and baseline tiers to OR'd policies, with decision_wait and num_traces controlling the completeness-versus-memory trade-off.

Why does tail sampling need a two-layer otel collector topology?

…the slowest edge cases.

That decision only works if one collector sees every span of a trace. With a single instance it does. Once you scale to several replicas behind an ordinary load balancer, spans from one trace land on different instances. Each instance makes its decision from a fragment. One keeps the error span, another drops the parent, and the backend stores partial traces that are worse than no trace at all.

The fix is a stateless routing layer in front of the stateful sampling layer:

  • SDKs send OTLP to a regular load balancer.
  • Load-balancing tier: collectors run the loadbalancing exporter, hashing on trace ID so every span of a trace reaches the same downstream instance. This tier also derives RED metrics (for example, with the span metrics connector) so metrics reflect 100% of traffic, not the sampled subset.
  • Sampling tier: collectors run the tail sampling processor and buffer whole traces.
  • Backends: sampled traces go to trace storage, and metrics go to the metrics store.

The exporter needs two things: routing_key set to trace ID, and a resolver (static list, DNS, or Kubernetes) that discovers sampling-tier instances. A minimal config follows below.

Resizing the sampling tier is the main operational risk. Adding or removing an instance changes the hash ring, so spans of in-flight traces can route to a new owner. Both owners then decide on fragments. Expect some inconsistent traces during every scale event. Avoid aggressive autoscaling of the sampling tier, and scale during low traffic.

# Load-balancing tier: stateless; receives all spans from SDKs via an ordinary L4/L7 LB.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

connectors:
  # Derive RED metrics here, before sampling, so they reflect 100% of traffic.
  spanmetrics: {}

exporters:
  loadbalancing:
    # Hash on trace ID so every span of a trace reaches the same sampling instance.
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true  # placeholder; use real TLS settings in production
    resolver:
      # A headless Service returns one A record per sampling-tier pod.
      # Each change to this set re-shapes the hash ring and splits in-flight traces.
      dns:
        hostname: otel-sampling-headless.observability.svc.cluster.local
        port: 4317
  otlp/metrics:
    endpoint: metrics-gateway.observability.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing, spanmetrics]
    metrics:
      receivers: [spanmetrics]
      exporters: [otlp/metrics]

A load-balancing tier collector config that routes spans by trace ID to sampling-tier pods discovered through a headless DNS name and derives span metrics from unsampled traffic.

How do you size the sampling tier and estimate cost savings before rollout?

size it for the peak you expect during an incident from the start. Incidents are when error and latency policies keep the most traces and when you can least afford dropped spans.

Memory

The collector buffers every span until the decision wait expires, so:

memory ≈ spans/sec × decision wait (s) × average in-memory span size

Illustrative estimate (assumed numbers, not a benchmark): 20,000 spans/s × 30 s × 2 KB ≈ 1.2 GB per instance's share of traffic. Double that for runtime overhead and bursts. Also set an explicit cap on the number of traces held in memory. Without a cap, a retry storm can push the process into an OOM kill and you lose every buffered trace. With a cap, the collector evicts old traces early, which is a smaller and easier-to-diagnose failure.

Kept fraction

kept ≈ errors ∪ slow + baseline × (1 − errors ∪ slow)

Count the overlap between errors and slow requests once. Illustrative estimate (assumed numbers, not a benchmark): with 1% errors, 2% slow and 0.5% that are both, errors ∪ slow is 2.5%. Add a 5% baseline of the remaining traffic: 2.5% + 5% × 97.5% ≈ 7.4%. At an assumed 2 TB/day of ingest, that leaves about 0.15 TB/day to store.

Gain Cost
Lower backend ingest and storage Extra compute and memory for buffering
Fewer low-value traces to search An extra network hop through the load-balancing layer
Retention you can predict A stateful tier to scale, monitor and upgrade

Error rates rise sharply during outages, so the kept fraction climbs exactly when load peaks. Replay a sample of your own production traffic through these formulas before committing to a budget.

When should you combine head and tail sampling, and what are the alternatives?

A light head sample at the SDK or edge is reasonable when collector ingest or cross-zone network transfer is the cost you are trying to cut. The rule that constrains it: a span dropped at the head never reaches the tail sampler, so no policy can bring it back. At head time nobody knows whether a request will fail. Head-sample only traffic you already understand, such as health checks and high-volume read endpoints. Send everything else through at 100%.

Every service must also make the same decision for a given trace. If services sample independently, the trace arrives with missing spans and does not show the full call path. Use a parent-based sampler so downstream services follow the sampled flag they receive. If you need ratio sampling, make the decision deterministic on trace ID so every service reaches the same result.

Approach Strength Cost or risk
Head sampling only Simple and cheap at every hop Rare errors and slow requests are mostly lost
Keep everything, short retention No sampling blind spots Ingest and storage still scale with traffic
Metrics and logs, no full traces Lowest cost No cross-service causality during an incident
Vendor or backend sampling Less pipeline to operate Policies and drop visibility move outside your control

Tail policies select failures you can detect from span data, such as error status or high latency. A request that returns 200 with wrong data matches neither rule, so only the baseline sample captures it. Keep that baseline large enough to investigate these silent failures.

How do you roll out and operate tail sampling safely?

Roll policies out in stages. Start the sampling tier with a keep-everything policy so you can confirm that load balancing and trace completeness work before any data is discarded. Then add one restrictive policy at a time and compare kept-span volume against your estimate. Tightening a single policy per change means a sudden gap in traces has one obvious suspect.

Record the effective sampling rate on every kept trace, for example as a span attribute naming the matching policy and its probability. Without it, the kept set looks like reality. Because errors are kept at 100% and successes at a few percent, a naive error-rate query over stored traces will greatly overstate failures. With the rate recorded, engineers and query tools can reweight results or restrict aggregates to the baseline policy. Compute SLOs from metrics, not from sampled traces.

The Collector emits telemetry about itself. Scrape it and alert on these signals:

Signal Why it matters Alert when
Collector memory Buffered traces live in memory until decided Approaching the memory limiter threshold
Refused or dropped spans Data loss before any policy runs Any sustained non-zero rate
Decision latency Late spans miss the decision and arrive orphaned Late-span count rises
Trace-limit pressure Old traces are evicted undecided when the buffer is full Evictions occur during spikes

Treat this tier as critical-path infrastructure. Give it a runbook that covers scaling out, falling back to keep-all or to head-only sampling, and checking how load is distributed across the second layer. Load-test it at incident-level traffic. Error storms are the moment it fills up, and they are also when you most need it to work.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to opentelemetry tail sampling cost, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.