What is OpenTelemetry tail sampling and how does it differ from head sampling?
Head sampling makes the keep-or-drop decision when the root span starts. The SDK applies a sampler, such as a fixed ratio based on the trace ID, and records the result in the sampled flag of the trace context. Downstream services read that flag and follow it, so every span in the trace is either recorded or skipped. The decision is cheap because it costs one comparison and needs no coordination between services. Its weakness is timing: the sampler runs before anything interesting has happened. A request that later times out in a database call, or returns a 500 from a third service, had the same odds of being kept as a healthy one.
Tail sampling moves the decision into the otel collector. The collector holds every span of a trace in memory for a configured wait period, then evaluates policies against the complete trace. A policy can check for an error status, a duration above a threshold, a specific attribute, or a probabilistic rate. The aim is to keep slow and failing traces while dropping most of the routine successful ones, which reduces storage and backend cost without losing the traces engineers actually open.
Tail sampling suits backend and platform teams that already run a collector tier and whose trace volume makes full retention too expensive.
| Head sampling | Tail sampling | |
|---|---|---|
| Decision point | Root span, in the SDK | Collector, after buffering |
| Information available | Trace ID, root attributes | All spans, status, duration |
| What it reduces | Network traffic, SDK overhead | Storage, backend ingest |
| Operational cost | Negligible | Memory, routing, extra collector tier |
How does the tail sampling processor work inside the otel collector?
The processor keys every incoming span by trace ID and appends it to an in-memory buffer for that trace. The first span of a new trace starts a timer. When the configured decision wait expires, the processor evaluates its policies once against everything collected so far. It then forwards the whole trace down the pipeline or drops it as a unit. Spans are never sampled individually, so a kept trace arrives at the backend with all the spans the instance had received when the decision fired.
Spans that show up after the decision are the edge case. Depending on configuration, the processor can keep a cache of recent decisions and apply the cached verdict to late spans. Without that cache, or once an entry has been evicted, the late span is effectively treated as a new trace. That produces a fragment that may be judged differently from the rest of its trace.
This design turns a normally stateless pipeline into a stateful one. A batch or attributes processor can restart and lose a few in-flight batches. A tail sampler that crashes or is rolled loses every trace it was holding, and the SDKs will not resend those spans.
The decision wait is therefore a direct trade-off. A longer wait catches slow downstream spans and async work, so traces are more complete. It also holds more traces in memory at once and delays export of every trace, including the ones you keep. A shorter wait reverses both effects.
The tail sampling processor and the load-balancing exporter are not part of the core collector build. Their option names and defaults have changed between releases, so check the documentation for the exact distribution and version you deploy.
How do you route every span of a trace to the same sampling collector?
Tail sampling is stateful: the instance that decides on a trace must have seen every span of it. The usual layout has two tiers:
Apps → agent/gateway collectors (loadbalancing exporter) → sampling collectors (tail_sampling) → backend
The first tier is stateless and can scale freely. It runs the load-balancing exporter with the routing key set to the trace ID. The exporter hashes each trace ID onto a ring of sampling-tier endpoints, so spans from different services and different agents still land on the same sampler. Endpoints come from a resolver: a static list, a DNS name that returns multiple A records, or, where your build supports it, the Kubernetes API. A minimal configuration example follows this section. Verify the exporter, resolver, and routing option names against your collector version.
A round-robin L4 or L7 balancer in front of the sampling tier looks like it works, and that is the danger. Each sampler receives a random subset of every trace's spans and makes its own decision on that fragment. Policies that depend on errors or latency see only part of the evidence. The result is traces with missing children or orphaned roots, and nothing reports an error.
Scaling or rescheduling the sampling tier changes ring membership. Consistent hashing limits how many trace IDs move, but traces in flight during the change are split across two instances. Each instance decides independently, so one half may be kept and the other dropped. Expect a short burst of partial traces on every deploy, autoscale event, or pod eviction. To limit how often this happens, keep the tier's size stable and avoid aggressive autoscaling.
# Tier 1: stateless agent/gateway collector.
# Routes every span to a sampling-tier instance chosen by trace ID.
# Verify option names against your collector version.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
loadbalancing:
# Hash on trace ID so all spans of one trace reach the same sampler
routing_key: traceID
protocol:
otlp:
timeout: 1s
tls:
insecure: true # placeholder; enable TLS in production
resolver:
# A headless Service returns one A record per sampling pod.
# The resolver re-queries DNS, so ring membership changes when pods do.
dns:
hostname: otel-sampler-headless.observability.svc.cluster.local
port: 4317
# Alternative: a fixed list of sampling-tier endpoints
# static:
# hostnames:
# - sampler-0.example.internal:4317
# - sampler-1.example.internal:4317
service:
pipelines:
traces:
receivers: [otlp]
# No tail_sampling here: decisions happen only in the stateful tier
exporters: [loadbalancing]
A minimal first-tier collector config that uses the load-balancing exporter to route spans by trace ID to a DNS-resolved sampling tier, with a static resolver shown as an alternative.
How should you order processor pipelines and write trace sampling policies?
Processor order
Order inside the traces pipeline determines what the sampler can see and how much data it has to hold:
memory_limiterfirst. It refuses data before the sampler buffers it, so a traffic spike causes backpressure instead of an out-of-memory kill.- Enrichment next. If a policy matches on tenant, environment, or a normalized route, add that attribute before the sampler. Attributes added afterward do not exist when the decision is made.
tail_sampling.batchlast. Batching before the sampler adds delay and regroups spans before the decision window starts. That makes it harder to predict whether late spans arrive inside the window. Batch only the traces you have decided to keep.
Policies built around goals
A practical policy set has four parts:
- a
status_codepolicy that keeps every trace with an error - a
latencypolicy for traces above your SLO threshold - a
string_attributepolicy for critical endpoints or tenants - a
probabilisticbaseline for everything else, at a low percentage
Top-level policies combine with OR: a trace is kept if any policy matches. That is often not what you intended. A broad attribute policy can match most traffic and quietly turn the baseline sample into "keep nearly everything." An and policy, such as "tenant is X and latency exceeds Y," narrows matches explicitly. A composite policy assigns each sub-policy a share of a total span rate, which puts a cap on output.
Misconfigured policies do not raise errors. They show up as missing traces or as an unexpectedly large bill. Test each policy against recorded traffic before you roll it out.
processors:
memory_limiter: # first: refuse data before the sampler buffers it
check_interval: 1s
limit_mib: 1500
spike_limit_mib: 300
resource: # enrichment the policies may match on
attributes:
- key: deployment.environment
value: production
action: upsert
tail_sampling:
decision_wait: 10s # how long to wait for late spans
num_traces: 50000 # traces held in memory at once
policies: # top-level policies are ORed
- name: keep-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-slow
type: latency
latency: {threshold_ms: 2000}
- name: critical-tenant-and-slowish # AND narrows the match
type: and
and:
and_sub_policy:
- name: tenant
type: string_attribute
string_attribute: {key: tenant.id, values: [tenant-acme]}
- name: over-500ms
type: latency
latency: {threshold_ms: 500}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 5}
batch: {} # batch only what was kept
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resource, tail_sampling, batch]
exporters: [otlp]
Collector config fragment (receivers and exporters omitted) ordering memory_limiter, enrichment, tail_sampling, then batch, with error, latency, AND-scoped tenant, and 5% baseline policies.
How much memory does tail sampling need, and what does it cost to run?
Buffered memory is roughly the span rate multiplied by how long each trace waits for a decision and by the in-memory size of each span:
buffered memory ≈ spans per second × decision wait (s) × average in-memory span size
In-memory size is usually larger than the span's size on the wire, because the collector also stores per-trace bookkeeping. Estimate it generously.
Illustrative estimate (assumed numbers, not a benchmark). Each row changes one input from the baseline.
| Scenario | Spans/s | Wait (s) | Span size | Buffered memory |
|---|---|---|---|---|
| Baseline | 20,000 | 30 | 1 KB | ~600 MB |
| Longer wait to catch slow traces | 20,000 | 90 | 1 KB | ~1.8 GB |
| Larger spans (more attributes) | 20,000 | 30 | 2 KB | ~1.2 GB |
| Double traffic | 40,000 | 30 | 1 KB | ~1.2 GB |
The wait is the input teams are most tempted to raise. Catching a trace that takes 80 seconds requires holding every trace for about that long, including the fast ones. Memory therefore grows with the slowest trace you want to keep, not the typical one.
That memory is only part of the operating cost. Each instance needs headroom above the estimate for traffic spikes and garbage collection. The tier is stateful, so scaling it changes which instance receives each trace ID and has to be done carefully. There is also policy configuration to maintain and replay tests to run.
Against those costs, the backend stores a fraction of the spans. More importantly, the error and slow traces you need during an incident are kept every time rather than depending on chance. For many teams, the engineering time that saves outweighs the cost of the extra memory.
When should you avoid tail sampling, and what are the alternatives?
Tail sampling is a poor fit in several cases:
- Long or open-ended traces. Batch jobs, async workers and message-queue flows can span minutes or hours. The processor decides after a fixed wait, so it judges partial traces and spans that arrive later fall outside that decision.
- Overhead problems. Every span still leaves the application and crosses the network to the collector. If CPU in your services or egress cost is the issue, tail sampling does not help.
- Small volumes. If keeping everything is affordable, a stateful sampling tier adds failure modes without saving anything.
The alternatives:
| Approach | Reduces app/network load | Keeps all errors | Operational cost |
|---|---|---|---|
| Head sampling only | Yes | No | Low |
| Light head ratio + tail sampling | Partly | Only within the head-sampled set | Medium |
| Tail sampling only | No | Yes | High |
| Metrics and logs, minimal tracing | Yes | Errors are visible as counts, not traces | Low |
When you combine head and tail sampling, the rates multiply per policy. Illustrative estimate: with a 25% SDK ratio and a probabilistic policy keeping 10% of normal traces, each stored normal trace stands for 1 / (0.25 × 0.10) = 40 real requests. Each error trace kept at 100% stands for 4. You can only scale counts back up if you know which rate applied, so record the effective probability on each kept trace. Better still, compute RED metrics before any sampling happens.
How do you verify the sampler and keep RED metrics accurate?
Watch the collector's own telemetry
The collector reports its own metrics, and these are the first place to look when sampling misbehaves. Track spans refused by the memory limiter, spans dropped or failed at export, process memory against your limit, and the tail sampling processor's decision counts per policy. Metric names change between releases, so confirm them against your build's internal telemetry output. If refusals rise as traffic grows, the decision wait is holding too many traces. If one policy accounts for almost every keep, its condition is broader than you intended.
Test policies before rollout
Send synthetic traces through a staging pipeline: one trace with an error span deep in the call tree, one just over your latency threshold, and one ordinary success. The error and slow traces should always arrive at the backend, and ordinary traffic should arrive at roughly the configured rate. After rollout, look for traces with missing parents or orphaned subtrees in the backend. These usually mean the load-balancing tier sent spans from one trace to different samplers, so investigate routing before you change any policy.
Derive RED metrics before the sampler
Errors kept at 100% while successes are kept at 10% make sampled error rates look many times higher than they are. Even with recorded probabilities, low-volume endpoints produce noisy counts. To avoid this, fan the receiver out to two trace pipelines. The first sends every span to a span-metrics connector, which feeds a metrics pipeline. The second runs the tail sampler and exports traces. Request rate, error rate and duration then come from complete data. If you prefer, use the application's own metrics instead. Connectors have been added and renamed over time, so check that your distribution includes the one you plan to use.
# Illustrative collector config: derive RED metrics from every span, then tail-sample traces.
# Verify that spanmetrics and tail_sampling exist in your distribution and version.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
memory_limiter:
check_interval: 1s
limit_mib: 1500
tail_sampling:
decision_wait: 10s
policies:
- name: keep-errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: keep-slow
type: latency
latency: {threshold_ms: 2000}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 10}
connectors:
spanmetrics: {} # turns unsampled spans into request, error and duration metrics
exporters:
otlp/traces:
endpoint: traces-backend.example.internal:4317
otlp/metrics:
endpoint: metrics-backend.example.internal:4317
service:
pipelines:
traces/all: # sees 100% of spans; feeds the connector only
receivers: [otlp]
processors: [memory_limiter]
exporters: [spanmetrics]
traces/sampled: # same input, tail-sampled before export
receivers: [otlp]
processors: [memory_limiter, tail_sampling]
exporters: [otlp/traces]
metrics/red:
receivers: [spanmetrics]
exporters: [otlp/metrics]
An illustrative collector config that sends every span to a span-metrics connector for unbiased RED metrics while a parallel pipeline tail-samples traces before export.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to opentelemetry tail sampling, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.