What is driving your log volume, and how do you measure it?

Collector-side volume reduction means filtering, sampling or trimming log records in the pipeline, after applications emit them but before they reach your storage backend. The reasoning is simple: the cheapest log is the one never shipped. You avoid paying for egress, ingest, indexing and retention on data nobody reads. This guide is for backend and platform engineers who own log pipelines or run noisy services. The work pays off when ingest bills are climbing, a few chatty services dominate traffic, or debug output has leaked into production.

Before writing any rule, find out where the bytes come from. In most fleets a small number of sources account for most of the volume. Group recent logs by three dimensions:

  • Service, to find which team owns the problem.
  • Log level, to spot DEBUG or INFO floods.
  • Normalized message template or logger name, to collapse "user 4821 logged in" and "user 9917 logged in" into one pattern.

Measure bytes, not line counts. Most backends bill by ingested volume, and line counts are misleading. A health-check line might be 80 bytes, while a single error with a stack trace and request dump can be 20 KB.

Metric What it shows Blind spot
Line count Event frequency, chattiness Hides large payloads
Bytes What you actually pay for Hides high-frequency small lines unless sorted
Bytes per line Payload bloat, oversized fields Needs the other two for context

Sort by total bytes first. The top few templates are where you should start if you want to reduce log volume costs.

#!/usr/bin/env python3
"""Rank log sources by total bytes from a JSON-lines sample export.
Usage: python3 -I log_volume.py sample.jsonl
Assumes each line has 'service', 'level' and 'message' fields.
"""
import json, re, sys
from collections import defaultdict

# Replace variable parts so "user 4821 logged in" and "user 9917 logged in" share one template.
PATTERNS = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b", re.I), "<uuid>"),
    (re.compile(r"\b[0-9a-f]{16,}\b", re.I), "<hex>"),
    (re.compile(r"\b\d+(\.\d+)?\b"), "<num>"),
]

def template(msg):
    for rx, token in PATTERNS:
        msg = rx.sub(token, msg)
    return msg[:120]  # first line of a stack trace is enough to identify it

stats = defaultdict(lambda: [0, 0])  # (service, level, template) -> [bytes, lines]
with open(sys.argv[1], "rb") as f:
    for raw in f:
        try:
            rec = json.loads(raw)
        except ValueError:
            continue
        if not isinstance(rec, dict):
            continue
        key = (rec.get("service", "unknown"), rec.get("level", "unknown"),
               template(str(rec.get("message", ""))))
        stats[key][0] += len(raw)  # raw bytes, which is what ingest is billed on
        stats[key][1] += 1

total = sum(b for b, _ in stats.values()) or 1
print(f"{'bytes':>12} {'share':>6} {'lines':>9} {'B/line':>7}  service | level | template")
# Sort by bytes, not line count, and show the top 20 offenders.
for (svc, lvl, tpl), (b, n) in sorted(stats.items(), key=lambda kv: kv[1][0], reverse=True)[:20]:
    print(f"{b:>12} {b / total:>6.1%} {n:>9} {b // n:>7}  {svc} | {lvl} | {tpl}")

A standard-library Python script that groups a JSON-lines log sample by service, level and normalized message template, then ranks the groups by total bytes, showing share, line count and bytes per line side by side.

Should you fix, drop, sample or trim a noisy log source?

Work through each top template in the same order, and stop at the first rule that applies:

Order Action Applies when Where
1 Fix You own the code emitting the line Application
2 Drop The line never has diagnostic value Collector
3 Sample The line repeats but sometimes matters Collector
4 Trim The line matters but its payload is bloated Collector

Fixing comes first because a collector rule only hides the cost. The application still formats and ships every line, and the rule has to be kept in step with code it doesn't control. Drop candidates are lines nobody will query during an incident: health checks, readiness probes, successful static asset requests and debug output left on in production. Sampling suits repetitive INFO lines, where the pattern is useful and individual copies are not. Trimming keeps the event and removes oversized fields.

Before any rule ships, define a compliance allowlist. Security, access, financial and audit events are never dropped or sampled, whatever their volume. Evaluate the allowlist before every other rule so that a broad match can't silently remove them.

Illustrative estimate, not a measured result. Assume 1 TB/day. Health-check access logs make up 30%, and INFO lines from one chatty service make up another 40%. Dropping the health checks removes 30%. Sampling the INFO lines at 10% keeps 4% of the total and removes 36%. That leaves about 1 − 0.30 − 0.36 = 34% of the original volume, roughly 340 GB/day.

The fixed order also saves engineering time. Each source gets one recorded decision with a reason, so nobody has to argue about it again at the next cost review.

How does dropping logs work in the OpenTelemetry Collector?

A Collector pipeline has three stages. Receivers accept data, for example by tailing files or listening for OTLP. Processors transform or discard it. Exporters send it to a backend. The processors list in a pipeline's service definition sets execution order, and each record passes through the processors in that sequence. Every processor that sees a record spends CPU on it, and every exporter that sends it spends network and queue capacity. A rule that drops a record should therefore run first, so no later stage handles data that will be discarded anyway.

The intended order is:

receiver → [ filter: drop ] → [ sampler ] → [ transform: trim ] → [ batch ] → exporter

The filter processor drops any log record that matches a condition. Conditions are written in OTTL and can test three kinds of data:

  • Severity: for example, severity_number below INFO.
  • Resource attributes: for example, a specific k8s.namespace.name or service.name.
  • Body or attribute content: for example, IsMatch(body, "GET /healthz").

The configuration example below follows this structure. Field names and nesting have changed across releases, so check the filter processor syntax against the documentation for your Collector version.

Regex matching on log bodies costs more than the other condition types. Each pattern scans every record, and unanchored or alternation-heavy patterns cost the most. On a node agent, that CPU use competes with the workloads on the same node. If matching falls behind ingest, the memory limiter starts refusing data and receivers push back toward the source, where buffers fill and logs can be lost. Match on severity or attributes where possible, and reserve body regex for cases that need it.

receivers:
  otlp:
    protocols:
      grpc:

processors:
  # Refuses data when the agent nears its memory ceiling; this is where backpressure starts.
  memory_limiter:
    check_interval: 1s
    limit_mib: 400
    spike_limit_mib: 100
  # Drop stage: a record is discarded if ANY condition is true.
  filter/drop:
    error_mode: ignore
    logs:
      log_record:
        # Severity: drop DEBUG/TRACE but keep records with no severity set.
        - 'severity_number != SEVERITY_NUMBER_UNSPECIFIED and severity_number < SEVERITY_NUMBER_INFO'
        # Resource attribute: a cheap equality check.
        - 'resource.attributes["k8s.namespace.name"] == "dev-sandbox"'
        # Body content: the regex runs on every record that reaches it, so keep it short.
        - 'IsMatch(body, "GET /healthz")'
  # Sample stage: keep roughly a quarter of what survives the filter.
  probabilistic_sampler:
    sampling_percentage: 25
  # Trim stage: remove a bulky attribute and cap attribute value length.
  transform/trim:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "http.request.header.cookie")
          - truncate_all(attributes, 256)
  batch:

exporters:
  otlphttp:
    endpoint:

service:
  pipelines:
    logs:
      receivers: [otlp]
      # Execution order: drop, then sample, then trim, then batch and export.
      processors: [memory_limiter, filter/drop, probabilistic_sampler, transform/trim, batch]
      exporters: [otlphttp]

A logs pipeline that drops by severity, resource attribute and body content before sampling, trimming and batching; check the filter and transform field names against the documentation for your Collector version.

Which log sampling strategy should you use?

Sampling keeps a fraction of matching records. It suits sources where individual lines are interchangeable but the aggregate still matters. The strategies differ in what they protect.

Strategy What it keeps Best for Main risk
Uniform probabilistic A fixed percentage of all records High-volume, homogeneous sources Rare events thinned at the same rate as routine ones
Per-key (route, tenant, message template) A separate rate per key Skewed traffic where a few keys dominate Key cardinality and rule maintenance
Severity-aware Every WARN and ERROR, a fraction of INFO and DEBUG Application logs in general A bug that only logs at INFO is still sampled away

Make the decision deterministic when logs must line up with sampled traces. Hash the trace ID and compare it against the rate. Every service and collector then reaches the same verdict for the same request, so you keep whole requests that match the traces you kept. This only works if every line carries the trace ID. Lines without it need a fallback rule, and keeping them is the safer default.

Discarded records are gone for good. As an illustrative estimate, a bug that surfaces in 50 requests a day, sampled at 1%, leaves about one kept occurrence every two days. Sampling each record independently also breaks request context. You might keep the error line but lose the INFO lines that explain how the request got there.

Throttling and rate limits work differently. They drop whatever exceeds a fixed budget, and volume spikes usually happen during an incident, when you most need the logs. If you add a rate limit, apply it after severity-aware rules so that WARN and ERROR stay exempt.

import hashlib

# WARN and above bypass sampling entirely.
ALWAYS_KEEP = {"WARN", "ERROR", "FATAL"}
# Illustrative rates. Every service and collector must share the same values.
SAMPLE_RATES = {"INFO": 0.10, "DEBUG": 0.01}


def trace_bucket(trace_id: str) -> float:
    # Map the trace ID to a stable value in [0, 1). The same ID always
    # lands in the same bucket, on any host and in any language that uses SHA-256.
    digest = hashlib.sha256(trace_id.lower().encode("ascii")).digest()
    return int.from_bytes(digest[:8], "big") / 2**64


def keep(record: dict) -> bool:
    level = str(record.get("severity", "")).upper()
    if level in ALWAYS_KEEP:
        return True
    rate = SAMPLE_RATES.get(level, 1.0)  # Unknown levels: keep.
    trace_id = record.get("trace_id")
    if not trace_id:
        return True  # No key to sample on, so default to keeping the line.
    # One bucket per request means its lines are kept or dropped together.
    # Because DEBUG's rate is below INFO's, kept DEBUG traces are a subset of kept INFO traces.
    return trace_bucket(trace_id) < rate


records = [
    {"severity": "INFO", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "cart loaded"},
    {"severity": "DEBUG", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "cache miss"},
    {"severity": "ERROR", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "msg": "payment timeout"},
    {"severity": "INFO", "msg": "startup complete"},  # No trace ID.
]

for r in records:
    print("keep" if keep(r) else "drop", r["severity"], r["msg"])

This severity-aware sampler always keeps WARN and above and samples INFO and DEBUG by hashing the trace ID, so every service drops or keeps the same requests, and it keeps lines that have no trace ID.

How do you do the same with Fluent Bit filters?

Fluent Bit handles the same three jobs with separate filters. Each filter runs on the records whose tag matches its Match pattern, and filters run in the order they appear in the configuration. That order determines what survives, so put the cheap, broad rules first.

  • Drop with grep. The grep filter keeps records whose field matches a regular expression (Regex) or removes them (Exclude). A typical first rule excludes records where level matches ^DEBUG$ or path matches ^/healthz. Apply this before any parsing or enrichment so you don't spend CPU on lines you will discard.
  • Trim with modify or record_modifier. modify can remove or rename keys. record_modifier can strip keys or keep only an allowlist. An allowlist is safer for chatty services because a new field that a developer adds does not reach your backend without someone deciding it should.
  • Cap with throttle. throttle limits records per time window. The limit applies to everything the filter's Match covers, so scope it narrowly with tags. As covered in the previous section, place it after the severity rules so errors are not the records that get cut.

Filter names and parameters change between releases. Check them against the documentation for the Fluent Bit version you run.

Running filters in the agent cuts network egress from every node, but every node then carries its own copy of the rules. A central collector tier gives you one place to change policy. Neither replaces fixing the logging call in the application, which is the only fix that also removes the cost of producing the log.

# Fluent Bit classic config. Filters run top to bottom.
# Verify filter names and keys against the docs for your version.

# 1. Drop: cheap, broad rules first, before any parsing or enrichment.
#    Separate grep filters, so a record matching either rule is dropped.
[FILTER]
    Name     grep
    Match    app.*
    Exclude  level ^DEBUG$

[FILTER]
    Name     grep
    Match    app.*
    Exclude  path ^/healthz

# 2. Trim: remove a heavy field and rename another on one service.
[FILTER]
    Name     modify
    Match    app.payments
    Remove   request_body
    Rename   msg message

# 2b. Trim with an allowlist for a chatty service. New fields are
#     dropped until someone adds them here.
[FILTER]
    Name           record_modifier
    Match          app.checkout
    Allowlist_key  timestamp
    Allowlist_key  level
    Allowlist_key  message
    Allowlist_key  trace_id

# 3. Cap: rate limit one narrowly scoped stream, after the severity rules.
#    Illustrative values: average of 500 records per interval, over 5 intervals.
[FILTER]
    Name      throttle
    Match     app.checkout
    Rate      500
    Window    5
    Interval  1s

An illustrative Fluent Bit pipeline that drops debug and health-check records with grep, trims fields with modify and a record_modifier allowlist, and then caps one stream's rate with throttle.

Why does trimming fields often save more than dropping lines?

Dropping only removes the lines you have decided are worthless. Trimming reduces the size of every line you keep, and kept lines are usually most of your volume.

Illustrative estimate (assumed numbers, not a benchmark). Assume an average record of 2,000 bytes: 200 bytes of message, 100 bytes of timestamp and severity, 1,200 bytes of Kubernetes metadata (pod labels, annotations, and node and namespace fields repeated on every line), and 500 bytes of serialized request body.

  • Dropping 20% of lines saves dropped_fraction × total = 20%.
  • Cutting metadata to 300 bytes and deleting the body saves (bytes_removed / avg_record_size) × kept_fraction = (900 + 500) / 2,000 × 100% = 70%.

Run the formulas against your own field sizes. The bytes usually sit in a few attributes.

The fields that are usually worth removing:

  • Duplicated Kubernetes metadata. Keep the labels you query on, such as namespace, workload and container. Delete annotations and controller hashes. If your backend stores resource attributes once per batch, check whether these fields are being copied into every record's attributes.
  • Full request and response bodies. Remove them by default, or truncate them to a fixed length so the start of the payload is still visible.
  • Stack traces repeated on every retry. Keep the full trace on the first attempt. On later attempts, log the error type and a short trace hash.

In the OpenTelemetry Collector, the transform processor handles both cases. delete_key removes a named attribute, and truncate_all caps the length of every string attribute. The example below shows both. When you choose a truncation limit, set it a little above your longest legitimate value. A limit set below that can cut off IDs that people search on.

When should you avoid collector rules, and how do you roll them out safely?

Collector rules treat the symptom. A rule that drops a debug line emitted thousands of times per second hides a logging bug the owning team never sees, so fix the source when you control the code. Save rules for third-party components, legacy services and emergency cost cuts.

Rules also drift. If someone rewords a message or renames a field, a match can silently stop working and the volume comes back. A loosened pattern can do the opposite and swallow real errors. Once rules are spread across hundreds of agents, nobody can say with confidence what is being removed.

Log-derived metrics and alerts are the easiest to break without noticing. Suppose you sample INFO at 10% but compute the error rate as errors divided by all lines. The rate jumps roughly tenfold with no change in the service. To keep sampled data honest, do one of two things:

  • Add an attribute that records the sample rate.
  • Emit per-rule counters of dropped and sampled records.

Then rescale dashboards using that rate, or move them to real metrics.

Rollout step Why it matters
Keep rules in version control with review Rules decide what data exists, so give them the same scrutiny as code
Start in count-only or shadow mode where supported, or route matches to a cheap sink You see what a rule would remove before the data is gone
Compare volume per rule before and after Catches rules that match too little or too much
Keep a per-service override to disable rules Restores full logs during an incident

The cost is real. Writing and reviewing rules takes time upfront, and every rule needs maintenance when log formats change. In return you get fewer surprise bills and faster searches over smaller indexes. A few rules aimed at your largest sources usually justify that cost. A long tail of small rules often does not.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce log volume costs, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.