What is driving your log volume, and how do you measure it?
Collector-side volume reduction means filtering, sampling or trimming log records in the pipeline, after applications emit them but before they reach your storage backend. The reasoning is simple: the cheapest log is the one never shipped. You avoid paying for egress, ingest, indexing and retention on data nobody reads. This guide is for backend and platform engineers who own log pipelines or run noisy services. The work pays off when ingest bills are climbing, a few chatty services dominate traffic, or debug output has leaked into production.
Before writing any rule, find out where the bytes come from. In most fleets a small number of sources account for most of the volume. Group recent logs by three dimensions:
- Service, to find which team owns the problem.
- Log level, to spot DEBUG or INFO floods.
- Normalized message template or logger name, to collapse "user 4821 logged in" and "user 9917 logged in" into one pattern.
Measure bytes, not line counts. Most backends bill by ingested volume, and line counts are misleading. A health-check line might be 80 bytes, while a single error with a stack trace and request dump can be 20 KB.
| Metric | What it shows | Blind spot |
|---|---|---|
| Line count | Event frequency, chattiness | Hides large payloads |
| Bytes | What you actually pay for | Hides high-frequency small lines unless sorted |
| Bytes per line | Payload bloat, oversized fields | Needs the other two for context |
Sort by total bytes first. The top few templates are where you should start if you want to reduce log volume costs.
#!/usr/bin/env python3
"""Rank log sources by total bytes from a JSON-lines sample export.
Usage: python3 -I log_volume.py sample.jsonl
Assumes each line has 'service', 'level' and 'message' fields.
"""
import json, re, sys
from collections import defaultdict
# Replace variable parts so "user 4821 logged in" and "user 9917 logged in" share one template.
PATTERNS = [
(re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.I), "<uuid>"),
(re.compile(r"\b[0-9a-f]{16,}\b", re.I), "<hex>"),
(re.compile(r"\b\d+(\.\d+)?\b"), "<num>"),
]
def template(msg):
for rx, token in PATTERNS:
msg = rx.sub(token, msg)
return msg[:120] # first line of a stack trace is enough to identify it
stats = defaultdict(lambda: [0, 0]) # (service, level, template) -> [bytes, lines]
with open(sys.argv[1], "rb") as f:
for raw in f:
try:
rec = json.loads(raw)
except ValueError:
continue
if not isinstance(rec, dict):
continue
key = (rec.get("service", "unknown"), rec.get("level", "unknown"),
template(str(rec.get("message", ""))))
stats[key][0] += len(raw) # raw bytes, which is what ingest is billed on
stats[key][1] += 1
total = sum(b for b, _ in stats.values()) or 1
print(f"{'bytes':>12} {'share':>6} {'lines':>9} {'B/line':>7} service | level | template")
# Sort by bytes, not line count, and show the top 20 offenders.
for (svc, lvl, tpl), (b, n) in sorted(stats.items(), key=lambda kv: kv[1][0], reverse=True)[:20]:
print(f"{b:>12} {b / total:>6.1%} {n:>9} {b // n:>7} {svc} | {lvl} | {tpl}")
A standard-library Python script that groups a JSON-lines log sample by service, level and normalized message template, then ranks the groups by total bytes, showing share, line count and bytes per line side by side.
Should you fix, drop, sample or trim a noisy log source?
Work through each top template in the same order, and stop at the first rule that applies:
| Order | Action | Applies when | Where |
|---|---|---|---|
| 1 | Fix | You own the code emitting the line | Application |
| 2 | Drop | The line never has diagnostic value | Collector |
| 3 | Sample | The line repeats but sometimes matters | Collector |
| 4 | Trim | The line matters but its payload is bloated | Collector |
Fixing comes first because a collector rule only hides the cost. The application still formats and ships every line, and the rule has to be kept in step with code it doesn't control. Drop candidates are lines nobody will query during an incident: health checks, readiness probes, successful static asset requests and debug output left on in production. Sampling suits repetitive INFO lines, where the pattern is useful and individual copies are not. Trimming keeps the event and removes oversized fields.
Before any rule ships, define a compliance allowlist. Security, access, financial and audit events are never dropped or sampled, whatever their volume. Evaluate the allowlist before every other rule so that a broad match can't silently remove them.
Illustrative estimate, not a measured result. Assume 1 TB/day. Health-check access logs make up 30%, and INFO lines from one chatty service make up another 40%. Dropping the health checks removes 30%. Sampling the INFO lines at 10% keeps 4% of the total and removes 36%. That leaves about 1 − 0.30 − 0.36 = 34% of the original volume, roughly 340 GB/day.
The fixed order also saves engineering time. Each source gets one recorded decision with a reason, so nobody has to argue about it again at the next cost review.
How does dropping logs work in the OpenTelemetry Collector?
A Collector pipeline has three stages. Receivers accept data, for example by tailing files or listening for OTLP. Processors transform or discard it. Exporters send it to a backend. The processors list in a pipeline's service definition sets execution order, and each record passes through the processors in that sequence. Every processor that sees a record spends CPU on it, and every exporter that sends it spends network and queue capacity. A rule that drops a record should therefore run first, so no later stage handles data that will be discarded anyway.
The intended order is:
receiver → [ filter: drop ] → [ sampler ] → [ transform: trim ] → [ batch ] → exporter
The filter processor drops any log record that matches a condition. Conditions are written in OTTL and can test three kinds of data:
- Severity: for example,
severity_numberbelow INFO. - Resource attributes: for example, a specific
k8s.namespace.nameorservice.name. - Body or attribute content: for example,
IsMatch(body, "GET /healthz").
The configuration example below follows this structure. Field names and nesting have changed across releases, so check the filter processor syntax against the documentation for your Collector version.
Regex matching on log bodies costs more than the other condition types. Each pattern scans every record, and unanchored or alternation-heavy patterns cost the most. On a node agent, that CPU use competes with the workloads on the same node. If matching falls behind ingest, the memory limiter starts refusing data and receivers push back toward the source, where buffers fill and logs can be lost. Match on severity or attributes where possible, and reserve body regex for cases that need it.
receivers:
otlp:
protocols:
grpc:
processors:
# Refuses data when the agent nears its memory ceiling; this is where backpressure starts.
memory_limiter:
check_interval: 1s
limit_mib: 400
spike_limit_mib: 100
# Drop stage: a record is discarded if ANY condition is true.
filter/drop:
error_mode: ignore
logs:
log_record:
# Severity: drop DEBUG/TRACE but keep records with no severity set.
- 'severity_number != SEVERITY_NUMBER_UNSPECIFIED and severity_number < SEVERITY_NUMBER_INFO'
# Resource attribute: a cheap equality check.
- 'resource.attributes["k8s.namespace.name"] == "dev-sandbox"'
# Body content: the regex runs on every record that reaches it, so keep it short.
- 'IsMatch(body, "GET /healthz")'
# Sample stage: keep roughly a quarter of what survives the filter.
probabilistic_sampler:
sampling_percentage: 25
# Trim stage: remove a bulky attribute and cap attribute value length.
transform/trim:
log_statements:
- context: log
statements:
- delete_key(attributes, "http.request.header.cookie")
- truncate_all(attributes, 256)
batch:
exporters:
otlphttp:
endpoint:
service:
pipelines:
logs:
receivers: [otlp]
# Execution order: drop, then sample, then trim, then batch and export.
processors: [memory_limiter, filter/drop, probabilistic_sampler, transform/trim, batch]
exporters: [otlphttp]
A logs pipeline that drops by severity, resource attribute and body content before sampling, trimming and batching; check the filter and transform field names against the documentation for your Collector version.
Which log sampling strategy should you use?
Sampling keeps a fraction of matching records. It suits sources where individual lines are interchangeable but the aggregate still matters. The strategies differ in what they protect.
| Strategy | What it keeps | Best for | Main risk |
|---|---|---|---|
| Uniform probabilistic | A fixed percentage of all records | High-volume, homogeneous sources | Rare events thinned at the same rate as routine ones |
| Per-key (route, tenant, message template) | A separate rate per key | Skewed traffic where a few keys dominate | Key cardinality and rule maintenance |
| Severity-aware | Every WARN and ERROR, a fraction of INFO and DEBUG | Application logs in general | A bug that only logs at INFO is still sampled away |
Make the decision deterministic when logs must line up with sampled traces. Hash the trace ID and compare it against the rate. Every service and collector then reaches the same verdict for the same request, so you keep whole requests that match the traces you kept. This only works if every line carries the trace ID. Lines without it need a fallback rule, and keeping them is the safer default.
Discarded records are gone for good. As an illustrative estimate, a bug that surfaces in 50 requests a day, sampled at 1%, leaves about one kept occurrence every two days. Sampling each record independently also breaks request context. You might keep the error line but lose the INFO lines that explain how the request got there.
Throttling and rate limits work differently. They drop whatever exceeds a fixed budget, and volume spikes usually happen during an incident, when you most need the logs. If you add a rate limit, apply it after severity-aware rules so that WARN and ERROR stay exempt.
import hashlib
# WARN and above bypass sampling entirely.
ALWAYS_KEEP = {"WARN", "ERROR", "FATAL"}
# Illustrative rates. Every service and collector must share the same values.
SAMPLE_RATES = {"INFO": 0.10, "DEBUG": 0.01}
def trace_bucket(trace_id: str) -> float:
# Map the trace ID to a stable value in [0, 1). The same ID always
# lands in the same bucket, on any host and in any language that uses SHA-256.
digest = hashlib.sha256(trace_id.lower().encode("ascii")).digest()
return int.from_bytes(digest[:8], "big") / 2**64
def keep(record: dict) -> bool:
level = str(record.get("severity", "")).upper()
if level in ALWAYS_KEEP:
return True
rate = SAMPLE_RATES.get(level, 1.0) # Unknown levels: keep.
trace_id = record.get("trace_id")
if not trace_id:
return True # No key to sample on, so default to keeping the line.
# One bucket per request means its lines are kept or dropped together.
# Because DEBUG's rate is below INFO's, kept DEBUG traces are a subset of kept INFO traces.
return trace_bucket(trace_id) < rate
records = [
{"severity": "INFO", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "cart loaded"},
{"severity": "DEBUG", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "cache miss"},
{"severity": "ERROR", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "payment timeout"},
{"severity": "INFO", "msg": "startup complete"}, # No trace ID.
]
for r in records:
print("keep" if keep(r) else "drop", r["severity"], r["msg"])
This severity-aware sampler always keeps WARN and above and samples INFO and DEBUG by hashing the trace ID, so every service drops or keeps the same requests, and it keeps lines that have no trace ID.
How do you do the same with Fluent Bit filters?
Fluent Bit handles the same three jobs with separate filters. Each filter runs on the records whose tag matches its Match pattern, and filters run in the order they appear in the configuration. That order determines what survives, so put the cheap, broad rules first.
- Drop with
grep. Thegrepfilter keeps records whose field matches a regular expression (Regex) or removes them (Exclude). A typical first rule excludes records wherelevelmatches^DEBUG$orpathmatches^/healthz. Apply this before any parsing or enrichment so you don't spend CPU on lines you will discard. - Trim with
modifyorrecord_modifier.modifycan remove or rename keys.record_modifiercan strip keys or keep only an allowlist. An allowlist is safer for chatty services because a new field that a developer adds does not reach your backend without someone deciding it should. - Cap with
throttle.throttlelimits records per time window. The limit applies to everything the filter'sMatchcovers, so scope it narrowly with tags. As covered in the previous section, place it after the severity rules so errors are not the records that get cut.
Filter names and parameters change between releases. Check them against the documentation for the Fluent Bit version you run.
Running filters in the agent cuts network egress from every node, but every node then carries its own copy of the rules. A central collector tier gives you one place to change policy. Neither replaces fixing the logging call in the application, which is the only fix that also removes the cost of producing the log.
# Fluent Bit classic config. Filters run top to bottom.
# Verify filter names and keys against the docs for your version.
# 1. Drop: cheap, broad rules first, before any parsing or enrichment.
# Separate grep filters, so a record matching either rule is dropped.
[FILTER]
Name grep
Match app.*
Exclude level ^DEBUG$
[FILTER]
Name grep
Match app.*
Exclude path ^/healthz
# 2. Trim: remove a heavy field and rename another on one service.
[FILTER]
Name modify
Match app.payments
Remove request_body
Rename msg message
# 2b. Trim with an allowlist for a chatty service. New fields are
# dropped until someone adds them here.
[FILTER]
Name record_modifier
Match app.checkout
Allowlist_key timestamp
Allowlist_key level
Allowlist_key message
Allowlist_key trace_id
# 3. Cap: rate limit one narrowly scoped stream, after the severity rules.
# Illustrative values: average of 500 records per interval, over 5 intervals.
[FILTER]
Name throttle
Match app.checkout
Rate 500
Window 5
Interval 1s
An illustrative Fluent Bit pipeline that drops debug and health-check records with grep, trims fields with modify and a record_modifier allowlist, and then caps one stream's rate with throttle.
Why does trimming fields often save more than dropping lines?
Dropping only removes the lines you have decided are worthless. Trimming reduces the size of every line you keep, and kept lines are usually most of your volume.
Illustrative estimate (assumed numbers, not a benchmark). Assume an average record of 2,000 bytes: 200 bytes of message, 100 bytes of timestamp and severity, 1,200 bytes of Kubernetes metadata (pod labels, annotations, and node and namespace fields repeated on every line), and 500 bytes of serialized request body.
- Dropping 20% of lines saves
dropped_fraction × total= 20%. - Cutting metadata to 300 bytes and deleting the body saves
(bytes_removed / avg_record_size) × kept_fraction= (900 + 500) / 2,000 × 100% = 70%.
Run the formulas against your own field sizes. The bytes usually sit in a few attributes.
The fields that are usually worth removing:
- Duplicated Kubernetes metadata. Keep the labels you query on, such as namespace, workload and container. Delete annotations and controller hashes. If your backend stores resource attributes once per batch, check whether these fields are being copied into every record's attributes.
- Full request and response bodies. Remove them by default, or truncate them to a fixed length so the start of the payload is still visible.
- Stack traces repeated on every retry. Keep the full trace on the first attempt. On later attempts, log the error type and a short trace hash.
In the OpenTelemetry Collector, the transform processor handles both cases. delete_key removes a named attribute, and truncate_all caps the length of every string attribute. The example below shows both. When you choose a truncation limit, set it a little above your longest legitimate value. A limit set below that can cut off IDs that people search on.
When should you avoid collector rules, and how do you roll them out safely?
Collector rules treat the symptom. A rule that drops a debug line emitted thousands of times per second hides a logging bug the owning team never sees, so fix the source when you control the code. Save rules for third-party components, legacy services and emergency cost cuts.
Rules also drift. If someone rewords a message or renames a field, a match can silently stop working and the volume comes back. A loosened pattern can do the opposite and swallow real errors. Once rules are spread across hundreds of agents, nobody can say with confidence what is being removed.
Log-derived metrics and alerts are the easiest to break without noticing. Suppose you sample INFO at 10% but compute the error rate as errors divided by all lines. The rate jumps roughly tenfold with no change in the service. To keep sampled data honest, do one of two things:
- Add an attribute that records the sample rate.
- Emit per-rule counters of dropped and sampled records.
Then rescale dashboards using that rate, or move them to real metrics.
| Rollout step | Why it matters |
|---|---|
| Keep rules in version control with review | Rules decide what data exists, so give them the same scrutiny as code |
| Start in count-only or shadow mode where supported, or route matches to a cheap sink | You see what a rule would remove before the data is gone |
| Compare volume per rule before and after | Catches rules that match too little or too much |
| Keep a per-service override to disable rules | Restores full logs during an incident |
The cost is real. Writing and reviewing rules takes time upfront, and every rule needs maintenance when log formats change. In return you get fewer surprise bills and faster searches over smaller indexes. A few rules aimed at your largest sources usually justify that cost. A long tail of small rules often does not.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to reduce log volume costs, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.