Tail Sampling in OpenTelemetry vs Head Sampling: A Cost Analysis

Where does OpenTelemetry tracing cost actually come from?

Tracing spend is several separate costs, and each one sits at a different point in the pipeline. Sampling decides which spans survive. Where the sampling decision happens determines which of these costs it actually reduces.

Sampling cuts how many spans you keep. It does not fix bloated spans. A span carrying a large request payload or a high-cardinality attribute such as a user ID is just as expensive when you keep 10% of traffic, so trim attributes before you tune sampling rates.

Illustrative estimate (assumed numbers, not a benchmark): monthly bytes = spans/sec × avg span bytes × seconds/month × kept fraction 2,000 spans/s × 1 KB × ~2.6M s ≈ 5.2 TB at 100% kept, or ≈ 0.52 TB at 10% kept.

Platform teams on ingest-based pricing feel this most directly, because their bill follows the last row of the table. Teams whose trace volume grows with traffic are in the same position: unless something upstream changes, cost rises with every new service and every new request.

How does sampling work in OpenTelemetry, end to end?

Sampling can happen at two points in the pipeline, and the two approaches differ in what information is available when the decision is made.

1. At span creation (head). When a service starts a span, the SDK's configured sampler returns one of three decisions: drop, record only, or record and sample. Only spans marked as sampled are handed to the exporter. The decision is written into the sampled flag of the W3C traceparent header, so downstream services can follow it.
2. In transit (collector). Exported spans travel over OTLP to one or more OpenTelemetry Collectors. A collector pipeline runs receivers, then processors in order, then exporters. Processors can filter, modify or sample spans before anything reaches the backend.
3. After the trace completes (tail). The tail sampling processor groups spans by trace ID, waits for the trace to finish arriving, and then applies policies that can inspect the whole trace: its status codes, its duration and its attributes.

Head sampling only knows the trace ID and whatever exists when the root span starts. Tail sampling knows how the request ended. Every cost and operational difference described below follows from that gap. The next two sections describe each mechanism in detail.

How does head sampling work, and when is it enough?

Head sampling cuts trace volume at the source. With the TraceIdRatioBased sampler, the root span's SDK derives a value from the trace ID and compares it against a threshold set by the configured ratio. Because the decision depends only on the trace ID, any service running the same calculation on the same trace reaches the same answer. The root records its decision in the sampled flag of the W3C traceparent header. Downstream services running the ParentBased sampler read that flag and follow it rather than deciding independently. As a result, a trace is either kept in full across every hop or dropped in full, and you avoid fragments with missing children.

This is the cheap, stateless default. The decision happens at span creation, before any attributes or events are recorded, so unsampled spans never consume exporter bandwidth, collector CPU or backend ingest. Nothing has to be buffered. The standard environment variables are enough to configure it: OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.1 keep roughly 10% of traces.

Head sampling is enough when:

• traffic is uniform,
• incidents can be diagnosed from metrics and logs, or
• you run very high-volume edges where any sample is statistically representative.

Avoid it when rare errors and tail-latency outliers are the traces you need during incidents. The decision is made before the outcome is known, so a 10% sample discards about 90% of failing requests along with the healthy ones.

Copyfrom opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

sampler = ParentBased(root=TraceIdRatioBased(0.1))

provider = TracerProvider(sampler=sampler)
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("checkout-service")

with tracer.start_as_current_span("handle_request") as root:
    # The decision is already made here, before any work or outcome is known.
    sampled = root.get_span_context().trace_flags.sampled
    with tracer.start_as_current_span("charge_card") as child:
        # The child inherits the root's decision rather than re-rolling.
        assert child.get_span_context().trace_flags.sampled == sampled
        # Unsampled spans are non-recording: attributes are dropped cheaply
        # and nothing reaches the exporter.
        child.set_attribute("payment.provider", "example-gateway")

print("trace sampled:", sampled)
provider.shutdown()

Configuring a parent-based trace ID ratio sampler in the OpenTelemetry Python SDK so the root span makes a 10% keep-or-drop decision that every child span inherits, with the equivalent environment variables shown in the header comment.

How does tail sampling work in the OpenTelemetry Collector?

Tail sampling moves the decision to after the trace has finished. The Collector's tail sampling processor holds incoming spans in memory, grouped by trace ID, until a configured decision_wait interval passes after the first span arrives. It then evaluates the complete trace against a list of policies:

• status_code keeps traces containing an error status.
• latency keeps traces whose duration exceeds a threshold.
• string_attribute / numeric_attribute match on values such as a route or a tenant ID.
• probabilistic keeps a fixed percentage.
• rate_limiting caps spans per second.

Top-level policies combine with OR semantics: if any policy votes to sample, the trace is kept. An and policy requires all of its sub-policies to match, for example slow and on the checkout route. A composite policy divides a rate budget across sub-policies so that one noisy rule cannot consume the entire allowance.

The configuration we use as a running example has three policies: a status_code policy matching ERROR, a latency policy with a threshold such as 500 ms, and a probabilistic policy at 5%. Errors and slow requests are always retained, and healthy traffic is sampled.

The processor can only judge a whole trace if it sees every span of that trace. With several Collector replicas behind a round-robin load balancer, spans from a single trace are spread across instances, and each instance makes its decision on a fragment. The standard fix is two tiers. The first is a stateless tier running the loadbalancing exporter with routing_key: traceID. It hashes each trace ID to a backend that is discovered through a static list, DNS, or a Kubernetes resolver. The second is a stateful tier that runs the tail sampling processor. Scaling that second tier changes the hash ring, so traces that are in flight during a scale event can still be split across instances.

Copyreceivers:
  otlp:
    protocols:
      grpc:
exporters:
  loadbalancing:
    routing_key: traceID        # hash trace ID -> one backend
    protocol:
      otlp:
        tls:
          insecure: true        # placeholder; use TLS in production
    resolver:
      dns:                      # headless Service in front of tier 2
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317
service:
  pipelines:
    traces: {receivers: [otlp], exporters: [loadbalancing]}
---
processors:
  tail_sampling:
    decision_wait: 10s          # wait after first span before evaluating
    num_traces: 50000           # max traces held in memory
    policies:                   # top-level policies are OR'd
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}
      - name: sample-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 5}
service:
  pipelines:                    # otlp receiver/exporter defined as usual
    traces: {receivers: [otlp], processors: [tail_sampling], exporters: [otlp]}

Two Collector configs: a stateless tier using the loadbalancing exporter to route spans by trace ID, and a tail-sampling tier that keeps all errors, all traces slower than 500 ms and 5% of the remaining traffic.

What does each sampling strategy cost? An illustrative estimate

Illustrative estimate (assumed numbers, not a benchmark). Assume 2,000 requests/s, 20 spans per trace, 500 bytes per span, $0.30 per GB ingested, a 1% error rate and a 2% slow-request rate with no overlap, over a 30-day month (2,592,000 s).

Monthly volume = RPS × spans per trace × span bytes × seconds = 2,000 × 20 × 500 × 2,592,000 ≈ 51,840 GB.

Tail keep fraction = errors + slow + 5% × (1 − errors − slow) = 0.01 + 0.02 + 0.05 × 0.97 = 7.85%.

Tail sampling moves cost out of the backend bill. Every span must reach the collectors before any decision is made, so 100% of the volume crosses the network. This example assumes $0.01/GB for that transfer, which is roughly what cross-zone traffic costs. It also assumes a stateful tier of six collectors at $150 per month each, sized for buffering memory.

At this moderate volume, tail sampling costs about $1,100 more per month than 10% head sampling, even though it sends fewer bytes to the backend. The fixed overhead of transfer and collectors outweighs the ingest savings. In exchange, you keep every error and every slow trace. Head sampling at 10% discards about 90% of those.

The comparison shifts as traffic grows. Ingest scales linearly with volume, while collector capacity scales in coarse steps. At higher volumes, the backend savings from tail sampling grow faster than the collector overhead. At low volumes, head sampling is usually cheaper.

How much memory does a tail-sampling collector need?

Memory scales with how much data the processor holds while it waits to decide:

memory ≈ span throughput × decision wait × average in-memory span size

Illustrative estimate, assuming 20,000 spans/s reaching the sampling tier, a 30-second decision_wait, and roughly 2 KB per span once deserialized (in-memory spans are larger than on the wire): 20,000 × 30 × 2 KB ≈ 1.2 GB of buffered spans. That is before runtime overhead and bursts, so plan for meaningfully more headroom.

decision_wait is the main lever. A short wait saves memory, but async work such as queue consumers or retries can arrive after the decision, which leaves fragmented traces. A long wait catches more late spans, but it raises memory linearly and delays when data reaches the backend. Long-running traces add to the problem because their spans stay buffered for the full window.

Also set num_traces above the expected number of concurrent traces. At about 10 spans per trace, that is 2,000 traces/s × 30 s = 60,000. When the processor reaches this limit, it evicts traces before deciding on them. expected_new_traces_per_sec helps it pre-size internal structures.

Put memory_limiter first in the pipeline, with limit_mib below the container limit and spike_limit_mib covering bursts. That way the collector refuses data before the OOM killer stops it.

This tier is stateful, which makes it hard to autoscale. The load-balancing exporter routes spans by trace ID. When you add a replica, remove one, or restart one, that mapping changes. In-flight traces split across instances, and spans buffered on a terminated pod are lost. Scale deliberately rather than on aggressive CPU triggers.

Copyreceivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2400        # hard limit, kept below the 3 GiB container limit
    spike_limit_mib: 600   # soft limit = 2400 - 600; refuse data above it
  tail_sampling:
    decision_wait: 30s                  # main memory lever: longer = more buffered spans
    num_traces: 100000                  # above ~60,000 concurrent traces, so no early eviction
    expected_new_traces_per_sec: 2000   # ~20,000 spans/s at ~10 spans per trace
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling]  # memory_limiter must run first
      exporters: [otlp]

An OpenTelemetry Collector config for the sampling tier, with memory_limiter placed first and tail_sampling sized from the section's illustrative inputs of 20,000 spans/s, a 30 s decision wait and about 2 KB per span.

What are the alternatives to pure head or tail sampling?

Most teams end up with a hybrid, because it attacks volume at the cheapest point in the pipeline.

Common hybrid patterns

• Drop noise first. Health checks, readiness probes and metrics-scrape spans rarely help in an incident. Filtering them in the collector pipeline before the tail sampler keeps them out of the buffer, which reduces both memory use and decision work.
• Light head sampling at high-volume edges. On a gateway handling heavy traffic, a modest head rate cuts what reaches the collectors at all. The tail sampler then chooses among the survivors.
• Always-sample overrides. Payment, login or other critical routes get a policy that keeps every trace on that path, whatever the baseline rate.

Rates compound. With an illustrative 10% head rate and a 20% tail rate for ordinary traffic, the effective rate is 2%. An error trace dropped at the head never reaches the tail sampler, so the error-keeping policy cannot recover it. Extrapolating request counts from sampled traces also gets harder, because each trace's weight depends on which policies it passed.

Copyreceivers:
  otlp:
    protocols:
      grpc:

processors:
  # Drop noisy spans before they enter the tail sampler's buffer.
  filter/drop-noise:
    error_mode: ignore
    traces:
      span:
        - 'attributes["url.path"] == "/healthz"'
        - 'attributes["url.path"] == "/readyz"'
  tail_sampling:
    decision_wait: 10s        # how long to buffer a trace before deciding
    num_traces: 50000         # traces held in memory; size to your traffic
    policies:                 # a trace is kept if ANY policy samples it
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: always-sample-critical-routes
        type: string_attribute
        string_attribute: {key: http.route, values: [/api/payments, /api/login]}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 20}

exporters:
  otlp:
    endpoint: tracing-backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/drop-noise, tail_sampling]  # filter runs first
      exporters: [otlp]

An OpenTelemetry Collector pipeline for a hybrid setup: noisy health-check spans are filtered before tail sampling, and the tail sampler keeps errors, always keeps critical routes, and samples 20% of the rest. Traffic that was already head-sampled at 10% at the edge ends up at an effective 2% rate.

Who should use tail sampling, and who should stick with head sampling?

Tail sampling suits teams that meet most of these conditions:

• Incidents hinge on rare traces. If postmortems regularly need the one failing request or the slow outlier, a head sampler will usually have discarded it.
• Many services take part in each request. Distributed traces across many hops are where a complete error trace saves the most investigation, and where per-service logs are hardest to stitch together.
• Volume is high enough for backend savings to matter. As the cost estimate shows, the fixed overhead of a collector tier only pays off once ingest is a large line item.
• Someone owns the collector tier. A stateful, memory-bound tier needs capacity planning, monitoring and careful scaling. A platform or observability team should own it, not each application team on its own.

Head sampling alone is the better fit for:

• small systems or a single service, where logs and metrics already explain most failures,
• low or uniform traffic, where ingest cost is modest and a random sample is representative, and
• teams without the capacity to run and monitor another stateful component.

If you fall in between, start with head sampling and good RED metrics. Add a tail-sampling tier once you find yourself missing error traces during incidents.

How can tail sampling save engineering time?

The cost tables only count infrastructure. Engineering time often matters more, and it shifts in both directions.

Where it saves time:

• Faster incident triage. With error and latency policies in place, the trace for a failing or slow request is stored. Engineers can go straight from an alert to the relevant trace instead of searching through samples, falling back to logs, or trying to reproduce the failure.
• Fewer "the trace was sampled out" dead ends. Under head sampling, the outcome of a request does not affect whether it is kept. Investigations of intermittent errors often stall because no trace exists. Policy-based retention removes most of that guesswork for the cases you define.
• Central policy changes. Head-sampling rates live in each service's SDK configuration. Changing them means touching many deployments. Tail-sampling policies live in collector configuration. One team can add an always-sample rule for a route under investigation, or tighten the baseline percentage, without redeploying application services.
• Cost tuning in one place. Volume questions such as "which service floods the backend?" and "what if we drop this route to 1%?" are answered and acted on at the collector rather than negotiated service by service.

Where it costs time:

• operating the stateful tier: sizing memory, tuning decision_wait, handling scale events and restarts,
• reviewing and pruning policies as they accumulate, and
• keeping metrics unbiased, as described in the next section.

The trade is usually worth it when incidents are frequent or expensive and a platform team already runs collectors. For a small system with rare incidents, the operational work can exceed the time saved during debugging.

How do you keep metrics honest and roll out tail sampling safely?

Tail sampling keeps errors and slow traces by design, so any query over stored traces is biased. If you keep every failed trace but 5% of successful ones, an error rate computed from stored spans reads roughly twenty times too high. Latency percentiles skew upward the same way. Dashboards and SLOs built on that data will page people for incidents that are not happening.

The fix is to compute RED metrics (rate, errors, duration) from data the sampler never touches. You can do this in two ways:

• Derive metrics from spans before sampling. In the collector, compute them in a pipeline stage that sees every span ahead of the tail-sampling processor.
• Use separate metrics instrumentation. Emit request counters and latency histograms from the services themselves.

Either way, alerts read from metrics, and traces become the evidence you open after an alert fires.

Rollout checklist

1. Measure span volume per service before changing anything. Without that baseline, you can't attribute savings or spot a service that floods the buffer.
2. Start conservatively. Use a high probabilistic percentage alongside your error and latency policies, so the first deployment changes little.
3. Watch the collector's own telemetry. Track dropped spans, refused data, and memory pressure. Any of these means decisions are being made on incomplete traces.
4. Tighten the percentage in steps. Hold each step long enough to cover normal traffic peaks.

Policies multiply over time. Give each one a named owner and review the set on a fixed schedule, deleting rules nobody can justify.

This modeling pays off during incidents. Engineers trust the dashboards instead of second-guessing the sample, and the budget reflects a rate you chose rather than one you find on the invoice.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to tail sampling opentelemetry, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.