Head vs Tail Sampling: Designing a Hybrid Trace Sampling Strategy

Head vs tail sampling: where each decision happens

Both strategies throw away most traces. They differ in when the keep-or-drop decision is made, and therefore in what information it can use.

Head sampling decides at the root span, before any work has happened. The SDK typically hashes or compares the trace ID against a configured ratio, then records the result in the trace context. With W3C Trace Context, this is the sampled bit in the traceparent trace-flags. Downstream services honour that flag, usually through a parent-based sampler, so a trace is either recorded everywhere or nowhere. The decision is cheap and stateless, but it is made blind. Nobody yet knows whether the request will fail or take ten seconds.

Tail sampling decides after spans have been emitted and gathered, usually in a collector tier that buffers each trace for a window. Because the trace is complete, or nearly complete, the policy can inspect span status, end-to-end latency, and attributes such as route or customer tier.

How head sampling works and why it is the default

Head sampling makes its decision when the root span starts, before any work is recorded. The decision travels downstream in the sampled flag of the W3C traceparent header, so every later service inherits it instead of deciding again.

Three sampler types cover most deployments:

• Parent-based samplers follow the incoming flag. If the root was sampled, every child span is recorded. If it was not, every child span is dropped. This keeps traces complete.
• Trace ID ratio samplers derive the decision deterministically from the trace ID. Any service that sees the same ID at the same ratio reaches the same answer, even without a parent decision to follow.
• Rate-limiting samplers, offered by some SDKs, cap the number of traces per second. This bounds volume during traffic spikes, which a fixed ratio does not.

All three are stateless and cost almost nothing per span. Head sampling is also the only approach that reduces load inside the application. Unsampled spans are never exported, so you save SDK serialization, export CPU and network egress. Tail sampling saves only backend cost.

A standard setup is OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1. The ratio applies only at roots, and child spans inherit the parent's decision.

The risk appears in mixed environments. Suppose one service uses an SDK configured with a plain ratio or always_on sampler, or a proxy strips traceparent. That service ignores the parent's flag and decides on its own, which produces fragmented traces or orphaned spans. Every language SDK and every hop must propagate the same context format and respect the parent's decision.

Copy# Shared sampling settings for every service, whatever its language SDK.
# Set these identically in each deployment so all hops agree.

# Use the parent's sampled flag when a traceparent is present.
# Apply the ratio only when this service starts a new root trace.
export OTEL_TRACES_SAMPLER=parentbased_traceidratio

# Keep roughly 10% of root traces. The decision is derived from the
# trace ID, so any service applying the same ratio gets the same answer.
export OTEL_TRACES_SAMPLER_ARG=0.1

# Propagate W3C Trace Context (traceparent) plus baggage. Every SDK and
# proxy in the path must read and forward the same format, or the
# downstream service starts a new root and the trace fragments.
export OTEL_PROPAGATORS=tracecontext,baggage

# Per-service values (placeholders).
export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=

# Avoid in mixed environments: these settings ignore the parent's flag.
# export OTEL_TRACES_SAMPLER=traceidratio   # re-decides at every hop
# export OTEL_TRACES_SAMPLER=always_on      # records children of unsampled roots

# Example inbound header. The final field holds the trace flags:
# 01 means sampled, 00 means not sampled.
# traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

These standard OpenTelemetry environment variables configure a parent-based trace ID ratio sampler with W3C Trace Context propagation. Applying them identically to every service keeps sampling decisions consistent across language SDKs.

Why head sampling misses the errors you need

Head sampling decides at the root span, before the request has done any work. The decision cannot depend on the outcome because the outcome does not exist yet. A failing request is kept at the same rate as a healthy one, so rare failures are thinned out at the sampling ratio.

Illustrative estimate

Assume these inputs:

• Request rate: 2,000 requests per second at the edge
• Error rate: 0.1%
• Head sampling ratio: 1%
• Slow path: 0.05% of requests take a slow path, such as a cache miss that falls through to a cold shard

Over one hour that is 7.2 million requests and 7,200 failures. At 1%, about 72 failing traces are kept and about 7,128 are discarded at the root. The slow path occurs about 3,600 times per hour, and about 36 of those traces survive. If the failure is limited to one tenant or one region, the kept sample can easily drop to zero.

The cost is uneven. The healthy traffic you keep is mostly redundant. The traces you most need during an incident are the ones you lose at roughly the same 99% rate.

What no sampler fixes

Tail sampling improves the odds for outcomes you can define in advance, such as errors and high latency. Any sampling still discards data for questions you did not anticipate. Exact error counts, latency percentiles, and SLO burn rates should come from metrics recorded before sampling. Forensic detail on a single rare request belongs in logs that carry the trace ID.

How tail sampling works in the OpenTelemetry Collector

The tail sampling processor holds every incoming span in memory, grouped by trace ID, for a fixed decision_wait period that starts when the first span of a trace arrives. When the timer expires, the processor evaluates the buffered trace against a list of policies. If any policy votes to sample, the whole trace is exported. Otherwise the whole trace is dropped. Because the decision is made on the assembled trace, it can use facts that a head sampler never sees: whether any span set an error status, how long the root took, or whether a span carries an attribute such as a tenant ID.

The common policy types are status_code, latency, string_attribute, probabilistic and rate_limiting. The and and composite policies combine or budget the others. A typical config has four policies:

• Error retention: an and policy that pairs status_code: ERROR with a rate limit, so an outage that fails every request cannot flood storage.
• Slow traces: a latency policy set somewhat above your p99.
• Baseline: a probabilistic policy at a low percentage, so normal traffic stays represented.
• Memory bound: num_traces caps how many traces are buffered at once.

The decision wait is a guess about how long a trace takes to finish. Spans that arrive after the decision are evaluated as a separate, incomplete trace or dropped outright. These include async consumers, retries and batch callbacks. Raising the wait reduces how often this happens, but it increases memory use roughly in proportion. The other cost is explanation: once you have a dozen overlapping policies, answering "why was this trace kept?" means reconstructing every evaluation, so keep the policy list short and name each policy clearly.

Copyprocessors:
  tail_sampling:
    # Buffer spans per trace ID this long after the first span arrives.
    # Spans that arrive after the decision miss the evaluation.
    decision_wait: 10s
    # Memory bound: maximum number of traces held in the buffer at once.
    num_traces: 50000
    policies:
      # Keep error traces, but cap retention so an outage that fails
      # every request cannot flood storage. status_code is listed first
      # so the rate limiter is meant to count only error traces.
      - name: errors-rate-limited
        type: and
        and:
          and_sub_policy:
            - name: is-error
              type: status_code
              status_code:
                status_codes: [ERROR]
            - name: error-budget
              type: rate_limiting
              rate_limiting:
                spans_per_second: 500
      # Keep slow traces. Set the threshold somewhat above your p99.
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 2000
      # Baseline so normal traffic stays represented.
      - name: baseline-1pct
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]

An annotated tail_sampling processor config that keeps rate-limited error traces, traces slower than 2 seconds and a 1% probabilistic baseline, with decision_wait and num_traces bounding the buffer (receiver and exporter definitions omitted).

Routing every span of a trace to the same collector

Tail sampling only works if one collector instance sees every span of a trace before its decision timer expires. Spans are produced by many services on many hosts, so a plain round-robin load balancer scatters them, and each sampler ends up judging fragments. The usual fix is a two-tier topology:

Copy [SDK] [SDK] [SDK]
    \    |    /
     v   v   v
 +-----------------------+   stateless, hash on trace ID
 | load-balancing tier   |
 +-----------------------+
     |       |       |
     v       v       v
 +------+ +------+ +------+  stateful, buffers spans
 |tail-1| |tail-2| |tail-3|  and applies policies
 +------+ +------+ +------+
     \       |       /
      v      v      v
        [ backend ]

The first tier runs the load-balancing exporter with its routing key set to the trace ID. It hashes each span onto a ring of sampling instances, which it discovers through a static list, DNS, or a Kubernetes resolver. Because the hash is deterministic, every span of a trace lands on the same sampler, no matter which SDK emitted it.

The weak point is membership. When autoscaling or a rolling deploy adds or removes a sampler, part of the ring remaps. Traces in flight are then split, with early spans buffered on one instance and late spans on another. Each instance decides independently. The result is partial traces in the backend, or two halves of one trace that get opposite decisions. To reduce this, scale the sampling tier slowly, drain instances for longer than the decision wait, and avoid scaling on short CPU spikes.

The topology also adds cost. It introduces an extra network hop, a second configuration to keep consistent, and a new failure domain: if the load-balancing tier stalls, every trace stalls with it.

Copy# Tier 1: stateless load-balancing collector (OpenTelemetry Collector, contrib distribution).
# It does no sampling; it only forwards each span to the tail-sampling
# instance that owns that span's trace ID.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317   # SDKs send OTLP here

exporters:
  loadbalancing:
    # Hash on trace ID so every span of a trace reaches the same sampler,
    # regardless of which service or host emitted it.
    routing_key: traceID
    protocol:
      otlp:
        timeout: 1s
        tls:
          insecure: true         # placeholder; enable TLS between tiers in production
    resolver:
      # Discover sampling instances via DNS. A headless Service returns one
      # A record per pod, and each IP becomes a member of the hash ring.
      # When this record set changes (scale-out, rolling deploy), part of
      # the ring remaps and in-flight traces can be split across samplers.
      dns:
        hostname: otel-tail-sampling-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      # No tail_sampling processor here; that runs only on tier 2.
      exporters: [loadbalancing]

A minimal load-balancing tier configuration that routes spans by trace ID to tail-sampling collectors discovered through DNS, with comments marking where membership changes cause traces to split.

Impact on tracing costs and engineering time

Sampling cost has three parts that scale differently: what you store, what you buffer, and what you transport. Write each as a formula before choosing a strategy.

• Storage and ingest = span rate × average span size × kept fraction × retention
• Tail buffer memory = span rate × average span size × decision wait
• Collector compute and network scale with the full span rate, not the kept fraction

Tail sampling only discards spans after they reach the collector. The SDK still exports every span, so SDK-to-collector traffic stays the same. Only the backend sees savings.

Illustrative estimate (assumed numbers, not a benchmark). The inputs are:

• 20,000 spans/s
• 1 KB average span
• 30 s decision wait
• 1% of traces contain an error

Full volume is 20,000 × 1 KB × 86,400 s ≈ 1.73 TB/day. The hybrid strategy head-samples 50% at the SDK, then tail-keeps all errors plus 10% of the remaining traces.

Buffer memory needs headroom on top of these figures. A traffic spike multiplies span rate, and therefore memory, for the full decision wait. An under-provisioned collector either refuses and drops spans or restarts after running out of memory, and a restart loses every trace still in its buffer.

Engineering time belongs in the same model. With head-only sampling, about 90% of failing requests leave no trace. Engineers then reconstruct the failure from logs or try to reproduce it without knowing what triggered it, which can take hours per incident. A retained error trace shows the failing path directly. Compare the collector capacity tail sampling requires against the engineering hours it saves on each incident.

Choosing a strategy: hybrid design, bias correction and alternatives

Head-only sampling fits services with low to moderate traffic, tight budgets, or teams that already detect errors through metrics and logs. If an alert fires from an error-rate metric and the logs carry a trace ID, a missing trace is an inconvenience rather than a blind spot. Tail or hybrid sampling fits high-traffic systems, teams with strict forensic requirements, and workloads where rare failures are expensive.

For most teams that need tail sampling, a hybrid works best. Apply moderate head sampling at the SDK, or none, to keep network egress and collector load predictable. Then run tail policies that keep every error, every trace above a latency threshold, and a small probabilistic baseline so normal traffic stays visible. Generate span-derived metrics, such as request counts and latency histograms, before any tail decision. That way dashboards and SLOs reflect all traffic, not only the traces you kept.

Sampling also biases your data. A tail sampler that keeps every error inflates the error rate seen in stored traces, and the stored latencies skew slow. Record the sampling probability or adjusted count wherever your tooling supports it. Never compute rates or percentiles from raw sampled counts. If exact counts matter, such as for billing or audits, take them from metrics and logs rather than from traces.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to head vs tail sampling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.