Why does dropping logs at the collector reduce log ingestion costs more than shortening retention?

Collector-side filtering means deciding, in the agent or collector that sits between your services and the logging backend, which records get forwarded. Dropping discards a record entirely. Sampling forwards only a fraction of records that match a rule. Either way, the bytes never reach the vendor.

Most log backends charge on several axes:

  • Ingestion: per GB received.
  • Indexing: per GB parsed and made searchable, often bundled with ingestion.
  • Retention: per GB stored per unit of time.

A record dropped at the collector avoids all three charges. Shortening retention only reduces the storage line, and storage is usually the cheapest of the three per GB.

Illustrative estimate (assumed numbers, not a benchmark)

These inputs are assumed:

  • Volume: D = 500 GB/day
  • Ingest and index price: P_ingest = $0.50/GB
  • Storage price: P_store = $0.10/GB-month
  • Retention: R = 30 days

The monthly cost is approximately:

D × 30 × P_ingest + D × R × P_store

Scenario Ingest cost Storage cost Monthly total Savings
Baseline (500 GB/day, 30 days) $7,500 $1,500 $9,000 —
Halve retention (15 days) $7,500 $750 $8,250 $750
Drop 30% at collector (350 GB/day, 30 days) $5,250 $1,050 $6,300 $2,700

Halving retention removes half of the smallest cost line. Cutting volume reduces every line.

Who should use collector-side filtering?

This approach is for platform and backend engineers who own pipelines or services with large log bills. It pays off most when a few streams make up most of the bytes, such as health checks, debug output, or chatty access logs. In that case, a handful of targeted rules can remove a large share of the volume.

It is a good fit when:

  • You run the collection layer. If your team already operates node agents or an aggregator tier, filtering is a config change in a system you control, not a new component.
  • Many teams ship logs through one pipeline. A central rule can remove the same noise, such as probe traffic or sidecar duplicates, from every service at once, without waiting for each team to change code.
  • Volume is concentrated. If the top few message templates account for most bytes, a few rules have a large effect. If volume is spread evenly across thousands of templates, rules give diminishing returns and fixing logging at the source is usually the better investment.

It is a poor fit when most of your log data is audit, security or compliance data that must be kept in full, or when total volume is already small enough that the savings would not cover the time spent maintaining rules.

How does collector-side filtering work?

A collector runs in one of two places, and many setups use both:

  • Node agent. A process on each host or Kubernetes node (often a DaemonSet) tails container or file logs and forwards them. Filtering here removes records before they cross the network, so it also saves node-to-backend bandwidth.
  • Aggregator tier. A smaller pool of collector instances receives logs from the agents before forwarding to the backend. Filtering here keeps rules in one place and can apply logic that needs a wider view than a single node, such as rate limits across all replicas of a service.

In either place, each record goes through the same steps:

  1. Read. An input plugin reads raw lines from files, the container runtime, a socket or a network protocol.
  2. Parse. A parser turns the raw line into a structured record with named fields such as level, service, path or trace_id. Filtering is only reliable on parsed fields.
  3. Evaluate rules in order. Each filter either passes the record unchanged, modifies it (removes or truncates fields), or drops it. Sampling rules make a keep-or-drop decision per record or per key.
  4. Buffer and forward. Surviving records are batched in memory or on disk and sent to the backend. Dropped records are never serialized for output and never reach the billing meter.

The rules are ordinary configuration. They are versioned, reviewed and rolled out like any other infrastructure config, which is what makes this lever fast compared with changing application code.

How do you find which logs are driving ingestion volume?

Before writing any filter rule, measure where the bytes come from. Group ingested logs by source (node, pod, or file path), service, log level, and message template. The template is the message with variable parts such as IDs, timestamps, and IPs replaced by placeholders. Grouping by template is what reveals that one log statement in one service accounts for a large share of the volume.

Use a window of at least several days that covers peak traffic and at least one deploy. A quiet afternoon underrepresents retry storms, and a window without deploys hides startup output.

Rank groups by bytes, not line count. Billing is usually based on ingested volume, so 500 lines carrying full request bodies can cost more than 50,000 short access lines. Ranking by count would hide them.

To compute bytes per group, take a sample of structured log records. For each record, measure the serialized length in bytes, not characters, so multibyte text is counted correctly. Add that length to a running total keyed by (service, level, template). Sort the totals in descending order and compute each group's share of the overall sum. The top ten groups are usually enough to show where filtering will pay off.

Category Why it is high volume Typical value
Health/readiness probe access logs Fires every few seconds per replica Near zero when the probe succeeds
DEBUG/TRACE in production Verbose output on every request path Low outside an active investigation
Repeated retry/connection-reset messages Bursts during an outage One sample per burst is enough
Chatty sidecars Proxy logs every request again Duplicates application logs
Framework startup banners Repeated on every restart Rarely read
Duplicated multiline stack traces Same trace on every occurrence The first occurrence plus a count is enough
import json
import re
import sys
from collections import Counter

PATTERNS = [
    (re.compile(r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"), "<uuid>"),
    (re.compile(r"\b\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?\b"), "<ip>"),
    (re.compile(r"\b0x[0-9a-fA-F]+\b"), "<hex>"),
    (re.compile(r"\b\d+(?:\.\d+)?\b"), "<num>"),
]

def template(msg):
    for pattern, placeholder in PATTERNS:
        msg = pattern.sub(placeholder, msg)
    return msg[:120]  # cap length so huge payloads do not create unique keys

bytes_by_group = Counter()
for line in sys.stdin:  # one JSON record per line (NDJSON sample)
    line = line.rstrip("\n")
    try:
        rec = json.loads(line)
    except ValueError:
        continue
    key = (rec.get("service", "?"), rec.get("level", "?"), template(str(rec.get("message", ""))))
    # Count encoded bytes of the record as shipped, not characters.
    bytes_by_group[key] += len(line.encode("utf-8"))

total = sum(bytes_by_group.values()) or 1
for (service, level, tmpl), size in bytes_by_group.most_common(10):
    print(f"{size / total:6.1%} {size:>12,d} B  {service:<16} {level:<6} {tmpl}")

A minimal Python script that reads a newline-delimited JSON log sample, reduces each message to a template, sums UTF-8 bytes per (service, level, template), and prints the ten largest groups by share of total volume.

Should you fix log levels in the application or filter at the collector?

Once you know which sources dominate volume, you can cut them in three places. Each one avoids a different share of the cost.

Option Costs avoided Time to deploy Main drawback
Fix log levels and logging hygiene in the application Generation, serialization, local disk, network, ingestion, storage Slow: a code or config change per service, plus review and deploy Needs time from every owning team
Drop or trim at the collector Ingestion and backend storage Fast: one central config change Still pays for generation, serialization, local disk and network to the collector
Backend retention tiers or archive-only storage Hot storage and indexing Fast Volume is unchanged, and per-GB ingestion charges usually still apply

The application fix is the only one that removes the work entirely. A debug statement that never runs costs no CPU for string formatting, no JSON encoding, no disk writes, and no bytes on the node-to-collector link. The other two options start paying only after all of that has happened.

Collector filtering has a second cost. It hides the misconfiguration instead of fixing it. A service shipping DEBUG in production usually has a wrong default log level. If you filter it out silently, that default gets copied into the next service built from the same template. Over time, the collector config fills with rules nobody can explain.

Use collector rules as a stopgap. Pair every drop rule with a ticket to the owning team, and put the ticket ID in a comment next to the rule. Remove the rule once the fix at the source ships. Backend tiers then handle what remains: logs you genuinely need, but rarely query.

What are the alternatives to collector-side filtering?

Collector filtering is one of several ways to cut log spend. The table above compares the three most common placements. The full set of options looks like this:

  • Fix logging at the source. Set correct default log levels, remove redundant log statements, and stop logging full payloads. This is the most complete fix, because the work is never done at all, but it needs a change in every service.
  • Sample in the application. Logging libraries and middleware can sample or rate-limit before a line is written. This saves serialization and local I/O as well, but each language and service has to implement it consistently.
  • Convert logs to metrics. If a log line exists only so someone can count events or measure latency, emit a counter or histogram instead. You keep the signal and lose the per-event bytes.
  • Use traces for request flows. Structured spans with sampling are often a better fit than log lines for following one request across services.
  • Backend retention tiers and archives. Keep less data in hot, indexed storage and move the rest to cheaper archive storage. This reduces storage cost but usually not ingestion cost.
  • Backend-side exclusion rules. Some backends let you discard data after it arrives. Check your pricing model before relying on this: if ingestion is billed on receipt, data discarded afterwards may still be charged.
  • Route to cheaper storage. Send rarely queried streams directly to object storage instead of the indexed backend, and load them only when needed.

These are not mutually exclusive. A common pattern is to filter at the collector for immediate savings, fix the source over time, and use tiers or archives for what remains.

How do Fluent Bit filters drop and slim down noisy logs?

Fluent Bit processes every record through a fixed sequence, and filters run after parsing and before anything leaves the node:

[input] ──▶ [parser] ──▶ [filter 1 … n] ──▶ [buffer] ──▶ [output]

Each record you drop here skips serialization and network transfer, and it never reaches the billing meter. Filters run in the order you declare them, so put the cheap drop rules first. That way the more expensive transformations only handle records that will actually be shipped.

Three tools, in order of cost

  • grep keeps or drops records whose field matches a regex. The configuration example that follows uses it to exclude records whose parsed request path is a health or readiness probe. It also uses record_modifier to remove a large field, such as a full request body or an embedded stack of headers.
  • modify / record_modifier rename, remove, or allowlist keys. Use them to remove Kubernetes metadata you already store as resource labels.
  • lua covers custom logic, such as keeping one in N records per key or redacting values conditionally.

Match on fields, not prose

Match rules against structured fields like level or path. Avoid matching against the free-text message. If a developer rewords a log line, a message regex can stop matching without any error, and the noise returns. A broad regex can also start dropping lines you need.

Every filter costs resources on every node

Complex regexes and Lua scripts consume CPU and memory on every node that runs the agent. When the filter chain can't keep up, records accumulate in the buffer. Once memory limits are reached, inputs pause or fall back to filesystem buffering. That backpressure can delay logs or drop them in ways nobody chose. Profile the agent under peak load before you add rules.

[SERVICE]
    Flush        1
    Parsers_File parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/app/*.log
    Tag           app.*
    # Parse JSON so the filters can match on structured keys, not the raw line
    Parser        json
    # Limit memory per input. When the limit is hit, this input pauses
    # (backpressure) instead of growing without bound.
    Mem_Buf_Limit 50MB

[FILTER]
    Name    grep
    Match   app.*
    Exclude path ^/(healthz|readyz|livez)$

[FILTER]
    Name       record_modifier
    Match      app.*
    Remove_key request_body

[OUTPUT]
    Name  forward
    Match app.*
    # Placeholder aggregator address
    Host  log-aggregator.internal
    Port  24224

A conservative Fluent Bit pipeline that uses grep to drop health and readiness probe records by their parsed path field, removes a large request_body field with record_modifier, and caps input buffer memory so backpressure stays bounded.

Which log sampling strategy should you use, and when should you avoid sampling?

Filtering removes a category of log entirely. Sampling keeps a fraction of it, so queries over that data return statistical answers instead of exact ones. Three strategies cover most cases:

Strategy How it decides Good fit Main risk
Uniform random Each record kept independently with probability p Repetitive success logs such as health checks and 200 responses Splits requests: line 1 of a trace survives, line 3 does not
Deterministic (hash-keyed) Hash of trace or request ID decides keep/drop Multi-line request flows across services Requires every line to carry the ID
Per-source rate limit Caps records per source per time window A single pod or job that bursts Drops whatever arrives after the cap is hit, errors included

Deterministic sampling works because every agent computes the same hash for the same ID. Related lines are kept or dropped together, and the agents do not need to coordinate.

Illustrative example. To keep 1 in 10 requests, hash trace_id with a stable function, take the result modulo 10, and keep the record when the remainder is 0. On each kept record, add sample_rate=10. Dashboards then sum sample_rate instead of counting rows, which recovers approximate true volume. A rate limiter's effective rate changes with load, so it cannot stamp a fixed value. Treat counts from throttled streams as lower bounds.

Avoid sampling in these cases:

  • Request-level debugging. You need the one failing request, not a representative sample.
  • Low-volume streams. The savings are negligible, and any gaps are obvious.
  • Audit and security data. Completeness is the purpose of this data, and it is often a compliance requirement.
  • Error bursts during incidents. A rate limit discards the errors you most need unless you exempt error and higher log levels from the cap.
import hashlib

SAMPLE_RATE = 10  # keep 1 in 10 requests
EXEMPT_LEVELS = {"ERROR", "FATAL"}  # never sample these away

def keep_bucket(trace_id: str, rate: int) -> bool:
    # Use a stable hash. Python's built-in hash() is salted per process,
    # so two agents would disagree. SHA-256 gives the same answer everywhere.
    digest = hashlib.sha256(trace_id.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big") % rate == 0

def sample(record: dict) -> dict | None:
    if record.get("level") in EXEMPT_LEVELS:
        record["sample_rate"] = 1  # kept unconditionally, counts as itself
        return record

    trace_id = record.get("trace_id")
    if not trace_id:
        # No key to hash on: keep rather than split an unknown request.
        record["sample_rate"] = 1
        return record

    if keep_bucket(trace_id, SAMPLE_RATE):
        # Each kept line stands in for SAMPLE_RATE lines.
        # Dashboards sum(sample_rate) instead of count(*).
        record["sample_rate"] = SAMPLE_RATE
        return record
    return None  # every line with this trace_id is dropped together

logs = [
    {"trace_id": "4bf92f3577b34da6", "level": "INFO", "msg": "GET /health 200"},
    {"trace_id": "4bf92f3577b34da6", "level": "INFO", "msg": "db query ok"},
    {"trace_id": "00f067aa0ba902b7", "level": "ERROR", "msg": "upstream timeout"},
]
kept = [r for r in (sample(dict(x)) for x in logs) if r is not None]
print(kept)
print("estimated volume:", sum(r["sample_rate"] for r in kept))

Deterministic sampling keyed on a SHA-256 hash of trace_id keeps or drops all lines of a request together. It stamps sample_rate on each kept record so counts can be re-weighted, and it always keeps ERROR and FATAL records.

What does a safe tiered log retention and sampling policy look like?

Before changing anything, review compliance, audit and legal retention obligations with whoever owns them. A regulation or contract that requires keeping a log category overrides any cost argument, and data dropped at the collector cannot be recovered later.

With that settled, assign each log category a tier based on how it is used:

Category Production handling Why
warn, error, fatal Keep 100% Low volume. Incident and debugging work depends on having all of them.
Security and audit events Keep 100%, separate route Gaps make the record useless as evidence.
info on hot success paths Sample, e.g. 1–10% Successful requests repeat the same pattern, so a sample shows behaviour well enough.
Other info Keep Usually modest volume and useful context.
debug Drop by default Highest volume, and rarely read outside an active investigation.

DEBUG still needs a way back in. Use a per-service override that expires automatically, for example one hour. If an override has no expiry, it tends to stay on after the incident ends, and the volume quietly returns.

Record count is only half the bill. You also pay for bytes per record:

  • Trim large fields. Truncate stack traces after the first frames on repeats, and cap or drop request and response bodies.
  • Avoid duplicated enrichment. If both the agent and the backend attach Kubernetes or host metadata, you pay for the same data twice on every line.
  • Convert to metrics. If a high-cardinality log line exists only so someone can ask "how many?", emit a counter instead. A metric series costs roughly the same whether it counts ten events or ten million.

How do you roll out log filtering without losing data you need?

Treat filtering as a production change, because a dropped log is gone. Unlike a retention cut, which you can reverse for future data, there is no replaying a line the collector never forwarded.

  1. Measure a baseline. Record daily volume per service, log level, and message pattern before touching anything. Without it, you cannot tell whether a rule saved money or simply coincided with a traffic dip.
  2. Run rules in count-only mode first. Where the collector allows it, tag matching records or route them to a cheap sink instead of discarding them. A week of counts shows what the rule would remove, and someone can check whether any of it was used in a recent incident.
  3. Emit dropped and sampled counts as pipeline metrics. Most collectors expose per-plugin record counts. Alert when a rule's drop rate shifts sharply. A sudden spike often means a new error is matching an old "noise" pattern.
  4. Give every rule an owner and a review date. If teams write their own rules with no central list, the rules drift. Patterns outlive the code that produced them, and nobody can answer "why are we missing these logs?"

The engineering payoff goes beyond the bill. Queries run faster, on-call engineers scroll through less noise, and finance raises fewer cost escalations. Weigh that against the ongoing work of maintaining rules, and against the real risk that an incident needs exactly the debug line you sampled away. Keep a fast path for temporarily disabling a rule during an investigation.

How can collector-side filtering save engineering time?

The time savings come from where the change happens and from what engineers no longer have to wade through:

  • One change instead of many. Removing probe logs or sidecar duplicates in every service means a code change, review and deploy per team. A collector rule does the same in a single reviewed config change, owned by the team that runs the pipeline. Product teams can then schedule the source fix instead of dropping other work for it.
  • Faster, cleaner searches. With less noise in the index, queries scan less data and return fewer irrelevant matches. During an incident, engineers spend less time writing exclusions to hide health checks and repeated retries before they can see the errors.
  • Fewer cost investigations. When volume is measured per template and controlled by named rules, a billing spike leads to a specific rule or a specific new log statement. Without that, someone has to reconstruct where the bytes came from after the invoice arrives.
  • Less pipeline firefighting. Lower volume means less pressure on agent buffers, aggregators and backend ingestion limits, so backpressure and throttling incidents become less frequent.

Illustrative estimate (assumed inputs, not a measurement). Assume 20 services each need the same probe-log fix, and each change takes an assumed 2 hours of engineering time across code, review and deploy. That is about 40 hours spread across 20 teams. A single collector rule, with an assumed 4 hours for writing, count-only testing and review, gets the cost reduction immediately, while the per-service fixes are scheduled over normal work. Your own numbers will differ. The point is that the collector change scales with the number of rules, not the number of services.

These savings only hold if the rules stay maintained. Rules without owners, review dates and drop-count metrics turn into a new source of investigation work, which is why the rollout steps above matter as much as the rules themselves.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce log ingestion costs, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.