What is log sampling and where does observability cost come from?
Log sampling means keeping a subset of log records, chosen by rules (drop health checks, keep all errors) or by probability (keep 10% of successful requests), and discarding the rest before ingest. You lower cost by removing records that tell you nothing new while keeping the ones you need during an incident.
In most backends, volume does not come from errors. It comes from success-path request logs, debug output left enabled in production, load-balancer and Kubernetes health checks, and chatty dependencies such as SDK retry messages or connection-pool logs.
Illustrative estimate (assumed numbers, not a benchmark)
Assumptions: a combined price of $0.60 per GB for ingest plus 30-day retention, and a 30-day month.
Monthly cost = Σ (daily GB × keep rate) × 30 × price per GB
| Source | Daily GB | Keep rate | Monthly before | Monthly after |
|---|---|---|---|---|
| Success-path requests | 120 | 10% | $2,160 | $216 |
| Debug logs | 80 | 5% | $1,440 | $72 |
| Health checks | 40 | 0% | $720 | $0 |
| Chatty dependencies | 50 | 25% | $900 | $225 |
| Errors and warnings | 10 | 100% | $180 | $180 |
| Total | 300 | $5,400 | $693 |
Error logs are about 3% of volume in this example, and they are the records you need most. That gap is why value per log should drive the decision, not raw volume. A thousand identical "200 OK" lines add almost nothing after the first few. A single timeout stack trace can be the only evidence of a failure. Set keep rates by asking how much each record type contributes to debugging, then let the cost follow from those rates.
What are the alternatives to random log sampling?
Random sampling is one option among six, and they differ mainly in where the keep-or-drop decision is made and how much context is available when it is made.
| Strategy | Cost reduction | State required | Error safety | Operational complexity |
|---|---|---|---|---|
| Rule-based dropping/filtering | High for known noise | None | High, if rules exclude errors | Low |
| Level-based routing to cheaper tiers | Storage cost only, ingest still paid | None | High, nothing is discarded | Low to medium |
| Uniform probabilistic sampling | Predictable, proportional | None | Low, errors dropped at the same rate | Low |
| Per-key rate limiting | High for bursty, repetitive sources | Counters per key | Medium, first N per key survive | Medium |
| Head-based sampling at the emitter | High, saves network and CPU too | None or minimal | Low, decided before outcome is known | Low |
| Tail-based sampling after completion | High | Buffers whole requests in memory | High | High |
Head sampling is cheap because it decides early, and that is also why it is unsafe for errors: the emitter cannot know a request will fail. Tail sampling sees the outcome, so it must hold every record until the request finishes. That brings memory pressure, timeout tuning and routing all of a request's records to the same instance. Per-key rate limiting falls between the two. It suppresses a log line that repeats ten thousand times a minute and still keeps its first occurrences.
Rule-based filtering is often the best option on its own. Health-check access logs, debug output left enabled in production, and duplicated framework banners have zero diagnostic value and are easy to identify. Dropping them takes no state and no coordination, and it never removes an error by chance. In many systems, filtering this known waste brings volume low enough that statistical sampling is not needed.
Why does uniform log sampling lose errors, and how does trace-ID keying fix it?
When volume still has to come down, the naive approach flips a coin for every log line: each line survives with probability p, independently of every other line. That causes two problems.
First, request context breaks. A request that emits 20 lines at a 10% rate keeps about two of them, picked at random. You see fragments of many requests and the full sequence of almost none, so you cannot reconstruct what happened.
Second, rare events disappear in proportion to the rate. Illustrative estimate: a service logs 40 errors an hour and samples at 5%. You keep about two per hour, and a failure that occurs once a day survives on roughly one day in twenty.
Trace-ID keying fixes the consistency problem. Instead of drawing a random number, hash the trace ID into a fixed range and keep the line if the result falls below a threshold. Every line carrying that trace ID gets the same decision, in every service that uses the same hash and threshold. A request is either kept whole or dropped whole.
This is head-based sampling: the decision depends only on an ID that exists when the request starts. It is cheap and stateless and needs no buffering. It also decides before the outcome is known, so errors must bypass it. Keep every line at error severity unconditionally, and accept that the surrounding context for those requests may still be dropped.
Logs without trace context, such as startup output, cron jobs and queue consumers that don't propagate context, need their own rules. Partial instrumentation produces inconsistent outcomes. A service that starts a new trace instead of continuing the incoming one hashes a different ID, so its half of the request gets a different decision.
import hashlib
import logging
import random
TRACE_RATE = 0.10 # keep 10% of traced requests
UNTRACED_RATE = 0.50 # separate rule for logs with no trace context
_SPACE = 2**64
def keep_trace(trace_id: str, rate: float = TRACE_RATE) -> bool:
# Deterministic: the same trace ID always maps to the same bucket, so every
# line of a request, in every service using this hash and rate, gets one decision.
digest = hashlib.sha256(trace_id.encode("ascii")).digest()
bucket = int.from_bytes(digest[:8], "big")
return bucket < rate * _SPACE
class TraceSampler(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
# Head sampling decides before the outcome is known, so errors bypass it.
if record.levelno >= logging.ERROR:
return True
trace_id = getattr(record, "trace_id", None)
if trace_id is None:
# Startup output, cron jobs, consumers without propagated context:
# no shared ID exists, so fall back to an explicit per-line rule.
return random.random() < UNTRACED_RATE
return keep_trace(trace_id)
handler = logging.StreamHandler()
handler.addFilter(TraceSampler())
log = logging.getLogger("checkout")
log.addHandler(handler)
log.setLevel(logging.INFO)
tid = "4bf92f3577b34da6a3ce929d0e0e4736" # placeholder trace ID
log.info("cart loaded", extra={"trace_id": tid}) # kept or dropped together
log.info("payment sent", extra={"trace_id": tid}) # same decision as above
log.error("payment declined", extra={"trace_id": tid}) # always kept
A Python logging filter that keys the keep-or-drop decision on a hash of the trace ID so a request's lines are kept or dropped together, lets error-level lines bypass sampling, and applies a separate rule to lines with no trace context.
How does tail sampling work?
Head-based sampling decides when a request starts, before anyone knows whether it will fail. Tail sampling defers that decision until the request is over. The Collector holds incoming spans, and logs correlated to them by trace ID, in memory for a fixed decision window. When the window closes, it evaluates the whole trace against a set of policies:
- Errors. Any span with an error status keeps the entire trace.
- Latency. End-to-end duration above a threshold keeps it.
- Attributes. Specific tenants, endpoints, or feature flags keep it.
Traces that match no policy are sampled at a low probabilistic rate. Because the unit of decision is the trace, you keep the healthy upstream calls that led to a failure along with the failure itself.
The costs are real:
- Memory and CPU. Every span and log for every in-flight trace sits in memory for the full window. Memory scales with throughput multiplied by window length, not with what you finally keep.
- Delivery latency. Nothing is exported until the window closes, so dashboards and alerts built on this data lag by at least that long.
- Premature decisions. Spans that arrive after the decision, such as long-running jobs, async consumers, or retries, are judged without the context that would have kept them. They may be dropped, or they may arrive as orphaned fragments.
- Co-location. All spans for a trace must reach the same Collector instance. A plain round-robin load balancer splits traces across instances, and no single instance sees enough of the trace to decide correctly. You need a routing tier that hashes on trace ID in front of the sampling tier. That makes scaling and failover harder, because adding or losing an instance remaps trace IDs, and traces in flight during the change are evaluated in pieces.
+-----------+ +-----------+ +-----------+
| Service A | | Service B | | Service C |
+-----+-----+ +-----+-----+ +-----+-----+
| | |
+---------------+---------------+ spans + logs w/ trace ID
|
v
+------------------------------------------------+
| Routing tier: route by hash(trace ID) |
| every span of a trace -> same Collector |
| add/lose instance = IDs remap mid-flight |
+----------+-------------------------+-----------+
| |
v v
+----------------------+ +----------------------+
| Sampling Collector 1 | | Sampling Collector N |
| buffer spans + logs | | same pipeline, | <- late span after
| by trace ID for the | | one shard of the | decision: dropped
| decision window | | trace-ID space | or orphaned
| then apply policies: | | |
| error? latency > T? | | |
| key attribute? | | |
+-----+----------+-----+ +----------+-----------+
| match | no match |
v keep all v keep low % v
+------------------------------------------------+
| Backend: export only after window closes (lag) |
+------------------------------------------------+
Tail sampling: a trace-ID-hashing routing tier sends every span of a trace to one Collector, which buffers spans and correlated logs for the decision window and then keeps whole traces that match error, latency, or attribute policies while sampling the rest at a low rate.
How do you implement log sampling in an OpenTelemetry Collector pipeline?
Treat the pipeline as layers, each discarding a different kind of waste.
Agent tier. Drop what never has diagnostic value before it leaves the host: health-check logs, debug output from noisy libraries, duplicate access lines. A filter processor handles exact drops. A probabilistic sampler on high-volume, low-value streams handles the rest. Before anything is dropped, use an attribute or transform processor to tag each record with its value: severity, route, tenant tier. Later stages then make decisions on fields instead of parsing message text.
Gateway tier. Route by trace ID with a load-balancing exporter so every span of a trace, and every log carrying that trace ID, reaches the same instance. Tail sampling then runs on traces. Typical policies are:
- Keep any trace with an error status.
- Keep traces above a latency threshold.
- Keep a small probabilistic baseline of everything else.
Keeping logs aligned. Most tail samplers act on traces only. Logs need a matching rule. The simplest option is deterministic sampling keyed on trace ID at the same ratio as the trace baseline, with error-severity logs always kept. Hash-consistent decisions keep the surviving logs and traces in agreement.
Metrics before sampling. Place a count or span-metrics connector ahead of any sampling stage so request rates and error rates reflect full traffic.
Recording the sample rate. Write the sampling probability onto each retained record. Weighting each record by the inverse of that probability restores accurate counts and percentiles at query time.
Processor and connector names and their options differ across Collector distributions and releases. Verify each one against the build you run.
# Illustrative gateway-tier sketch: verify names/options against your Collector build.
receivers:
otlp: {protocols: {grpc: {}}} # fed by agents via a traceID-routed loadbalancing exporter
connectors:
spanmetrics: {} # rate/error/duration metrics from ALL spans, before sampling
count: {} # log record counts from ALL logs, before sampling
processors:
filter/noise: # exact drops with no diagnostic value
logs:
log_record:
- 'attributes["http.route"] == "/healthz"'
transform/tag: # tag value and weight before the sampling decision
log_statements:
- context: log
statements:
- set(attributes["sample.rate"], 10) # inverse of 10% keep probability
- set(attributes["sampling.priority"], 100) where severity_number >= SEVERITY_NUMBER_ERROR
- set(attributes["sample.rate"], 1) where severity_number >= SEVERITY_NUMBER_ERROR
probabilistic_sampler/logs:
sampling_percentage: 10 # same ratio as the trace baseline
attribute_source: traceID # hash on trace ID so logs follow traces
sampling_priority: sampling.priority # 100 = always keep (errors)
tail_sampling:
decision_wait: 10s
policies:
- {name: errors, type: status_code, status_code: {status_codes: [ERROR]}}
- {name: slow, type: latency, latency: {threshold_ms: 2000}}
- {name: baseline, type: probabilistic, probabilistic: {sampling_percentage: 10}}
exporters:
otlphttp: {endpoint:
service:
pipelines:
traces/metrics: {receivers: [otlp], exporters: [spanmetrics]}
traces: {receivers: [otlp], processors: [tail_sampling], exporters: [otlphttp]}
logs/metrics: {receivers: [otlp], exporters: [count]}
logs: {receivers: [otlp], processors: [filter/noise, transform/tag, probabilistic_sampler/logs], exporters: [otlphttp]}
metrics: {receivers: [spanmetrics, count], exporters: [otlphttp]}
Illustrative gateway config: metrics are derived from full traffic, traces are tail-sampled, and logs are hash-sampled on trace ID with errors always kept and a sample.rate weight attached for re-weighting.
What should never be sampled, and when should you avoid log sampling?
Some logs have to bypass the sampler entirely. Keep errors and warnings, security and audit events (logins, permission changes, admin actions), and any log tied to a state change such as a payment, refund, or credential update. These are the records you need when you reconstruct an incident, and a 1-in-10 chance of having the one that matters is not acceptable.
Keep-rules alone are not enough. A single noisy endpoint or tenant can fill the kept budget and push everyone else's logs out. Apply per-route and per-tenant rate limits after the keep-rules, so one runaway client throws away its own excess instead of other tenants' data.
| Situation | Sample? | Reason |
|---|---|---|
| High-volume, healthy request logs | Yes | Redundant, and cheap to estimate from a fraction |
| Low-volume services | No | Little to save, and every line is signal |
| Compliance-bound logs | No | Retention rules usually require completeness |
| Bugs that return success or log at info | No | Level-based keep-rules never fire on them |
The last row is the dangerous one. If a service returns 200 with wrong data, or logs a failure at info level, your error keep-rule reports full coverage while the evidence gets dropped. Debugging a customer-specific issue that never raised an error then becomes a long round trip: you ask the customer to reproduce it, enable full logging for that tenant, and wait. Per-tenant overrides and on-demand full logging fix this, but each one adds configuration to maintain and test.
Sampling also reduces ingest without necessarily reducing total cost. Agents still collect and serialize every line, Collectors spend CPU making decisions, and tail sampling holds spans in memory until each decision is made.
How do you roll out log sampling safely and save engineering time?
Start by measuring log volume per service, logger, and severity before writing any sampling rule. Volume is usually concentrated in a few sources, so this tells you where sampling actually pays off. Next, remove obvious waste: health-check and readiness-probe logs, duplicate lines emitted by both a library and its caller, and debug output left on in production. Dropping these is safe and needs no statistics.
Validate before you commit
Run sampling in shadow or dual-write mode. Keep sending full data to its current destination, and route the sampled stream somewhere you can query separately. Then replay two or three past incident investigations against the sampled data. If an engineer could not have found the root cause using only the sampled stream, tighten the rules. Next, audit every dashboard and log-based alert. Counts derived from sampled data must be re-weighted by the sampling rate, or the numbers will silently fall. Finally, add an alert on unexpected drops in error log counts, so a misconfigured rule is caught before you need the data in an incident.
Costs and payoff
This is real development work. You add a pipeline tier, pay for Collector compute and memory, and take on rules that must be maintained as services change. The return is largest for high-volume services that already have mature tracing, because trace-ID keying and tail decisions depend on consistent context propagation.
Engineering time is saved in three places:
- Queries run faster because there is less noise to scan.
- There are fewer cost-driven firefights when an ingest bill spikes.
- Sampling policy lives in central Collector rules, instead of requiring logging changes in each service.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to log sampling, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.