How to Reduce Logging Costs with Retention Tiers and Ingest Filtering

Where does logging spend actually go?

Most log platforms bill along four axes, and they do not scale the same way.

Ingest usually dominates because you pay it once for every byte, including debug lines and health checks that nobody ever reads. Indexing also costs CPU and builds index structures on top of the raw data. Hot storage multiplies that same volume by the number of retention days. Archive storage is cheap per GB, and query cost depends on how people use the system, not on how much you send.

Illustrative estimate (assumed numbers, not a benchmark). Assume 500 GB/day, $0.50/GB ingest, 15 days hot at $0.10/GB-month, and 365 days archived at $0.02/GB-month with 10:1 compression.

Monthly cost ≈ D×30×P_ingest + D×R_hot×P_hot + (D×R_archive÷C)×P_archive

• Ingest: 500×30×0.50 = $7,500
• Hot: 500×15×0.10 = $750
• Archive: (500×365÷10)×0.02 = $365

Ingest is about 87% of the $8,615 total. Dropping 30% of bytes before indexing saves roughly $2,475 a month. That is ingest-time cost reduction: you filter, sample, or reroute data before the per-GB charge applies, so the savings carry through every tier after it.

How do you find the noisiest log sources?

Before writing any drop rule, measure where the bytes come from. Most log backends can report ingested volume grouped by a field. Group by three dimensions, in this order:

• Service or source. This shows which team owns the problem.
• Log level. This shows whether DEBUG or INFO output is running in production.
• Message template. This is the log line with variable parts stripped, such as "GET /healthz 200 in {n}ms". It shows which single statement is responsible.

Sort by bytes, not event count. A short heartbeat line logged constantly and a stack trace logged occasionally can cost the same.

The usual offenders are predictable:

• Health checks. Load balancers and orchestrators probe every instance on a fixed interval, all day.
• Debug logging. Someone enabled it during an incident and never turned it off.
• Per-request access logs. These duplicate data that metrics already capture.
• Retry loops. They log the same failure on every attempt, so volume spikes exactly when the system is degraded.

Volume concentrates because log output scales with call frequency, not importance. One statement on a hot path can outweigh hundreds of statements that fire rarely. A short list of templates therefore tends to dominate the total.

A platform engineer with access to billing and the log backend should run this analysis first. The owning service teams then confirm what is safe to cut. Doing this before writing rules matters: the result is a handful of targeted filters instead of weeks spent tuning low-volume sources that barely affect the bill. It pays off fastest when spend has grown faster than traffic, or just before a contract renewal.

Copy#!/usr/bin/env python3
# Usage: python3 -I log_volume.py sample.jsonl
# Ranks bytes by (service, level, message template) from a JSON-lines log export.
import json, re, sys
from collections import Counter

# Strip variable parts so identical log statements collapse into one template.
PATTERNS = [
    (re.compile(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", re.I), "{uuid}"),
    (re.compile(r"0x[0-9a-f]+", re.I), "{hex}"),
    (re.compile(r"\d+(?:\.\d+)?"), "{n}"),
]

def template(msg):
    for pattern, token in PATTERNS:
        msg = pattern.sub(token, msg)
    return msg

bytes_by_key = Counter()
total = 0
with open(sys.argv[1], "rb") as f:
    for raw in f:
        size = len(raw)  # measure bytes, not event count
        total += size
        try:
            event = json.loads(raw)
        except ValueError:
            bytes_by_key[("unparsed", "-", "-")] += size
            continue
        service = event.get("service", "unknown")
        level = str(event.get("level", "unknown")).upper()
        bytes_by_key[(service, level, template(str(event.get("message", ""))))] += size

# Top 20 templates with their share and cumulative share of total bytes.
running = 0
for (service, level, tmpl), size in bytes_by_key.most_common(20):
    running += size
    print(f"{size / max(total, 1):6.1%} {running / max(total, 1):6.1%}  {service:<16} {level:<6} {tmpl[:80]}")

A standard-library Python script that groups a JSON-lines log sample by service, level, and normalized message template, then prints the top templates by byte share and cumulative share. The cumulative column shows how few templates make up most of the volume.

How should you set log retention per log class?

Once you know which sources dominate volume, retention can follow from what each log class is actually used for. A single global retention period is usually wrong in both directions. It holds debug noise far longer than anyone reads it, and it may keep audit records for less time than required.

Check compliance first. Regulatory, contractual, or internal audit requirements can set a minimum retention period, require tamper-evident storage, or forbid sampling and dropping certain events entirely. Those requirements are fixed constraints. Every other retention period is a cost decision you can tune.

The ranges above are starting points, not recommendations for your environment.

Archives are cheap to store but slow to query. Rehydrating archived logs means re-ingesting or re-indexing them, which takes time and usually costs money per volume restored. Rehydration suits audits and late-discovered incidents. It does not suit an outage in progress. If on-call engineers regularly need data from the last two weeks, keep those two weeks hot.

Per-class retention also has a cost of its own. Every class adds routing rules, index or bucket policies, and another place where a log might live. Engineers lose time when they cannot tell whether a log was dropped, aged out, or archived. To limit this, keep the number of classes small, tag each event with its class at ingest, and document where each class lives and for how long.

How does a Vector pipeline filter and route logs before indexing?

Vector is the natural place to apply those class tags, because it processes every event before anything is indexed or billed. A pipeline has three kinds of components. Sources receive events, for example from files, journald, Kubernetes, or a syslog listener. Transforms inspect and modify them. Sinks deliver them to storage. Each event passes through the transforms in order, so a drop or trim applied early reduces the work and cost of every later stage.

Copy [kubernetes]  [journald]  [syslog]        sources
       \           |          /
        v          v          v
     [filter: drop debug, health checks]   transform
                   |
     [remap: delete/truncate fields, tag class]
                   |
              [route by class]
             /        |        \
            v         v         v
   [indexed store] [object storage] [discard]   sinks
     (hot, short)   (cheap, long)

A filter transform evaluates a condition and passes only the events that match. Typical uses are dropping debug level logs outside a canary, or discarding load balancer health check requests by matching on the source or the path.

A remap transform runs VRL, Vector's expression language. With del() you can remove fields nobody queries, such as full request headers or serialized payloads. With truncate() you can cap stack traces or message bodies at a fixed length. This step matters because indexed stores usually charge by ingested bytes, so removing a large unused field cuts cost on every event that carries it.

A route transform splits the stream by class. Logs used for debugging go to the indexed store with short retention. Audit and compliance logs go to object storage, where keeping data for years is inexpensive.

Copy# vector.yaml: filter, trim, and route logs by class before indexing
sources:
  k8s:
    type: kubernetes_logs   # assumes apps log JSON and fields are parsed upstream

transforms:
  drop_noise:
    type: filter
    inputs: [k8s]
    # Keep only events that are not debug and not health checks
    condition: '.level != "debug" && .path != "/healthz"'

  trim_and_tag:
    type: remap
    inputs: [drop_noise]
    source: |
      del(.request_headers)          # large, never queried
      del(.payload)
      if is_string(.stack_trace) {
        .stack_trace = truncate(string!(.stack_trace), 4096)
      }
      .log_class = "operational"
      if .kubernetes.container_name == "audit-logger" { .log_class = "audit" }

  by_class:
    type: route
    inputs: [trim_and_tag]
    route:
      operational: '.log_class == "operational"'
      audit: '.log_class == "audit"'

sinks:
  hot_index:                         # short retention, searchable
    type: elasticsearch
    inputs: [by_class.operational]
    endpoints: [""]

  archive:                           # cheap, long retention
    type: aws_s3
    inputs: [by_class.audit]
    bucket: example-audit-logs
    region: us-east-1
    compression: gzip
    encoding:
      codec: json

A Vector config that drops debug and health check events, deletes and truncates large fields, tags each event with a class, and routes operational logs to an indexed store and audit logs to S3.

When should you use log sampling, and when should you avoid it?

Sampling works when a log class has high volume, uniform content, and little value per line. Health-check responses and successful 200 request logs from a busy API are typical examples. Keeping 1 in 10 of those lines still shows latency shape and traffic patterns, and it cuts that class's ingest by roughly 90%.

Apply the sample only to that class. Errors, warnings, and audit events should skip the sampler entirely. These are the lines you search for during an incident, and they are already a small share of total volume, so sampling them saves little and costs a lot.

Key the sample on trace ID or request ID instead of choosing lines at random. With a hash-based key, every line for a given request is either kept or dropped as a group. An engineer who opens a sampled request then sees its full path through the services, not a scattered half of it.

Write the sample rate into each kept event as a field, such as sample_rate: 10. Without it, a dashboard that counts lines undercounts by an unknown factor. With it, queries can multiply counts back to estimates, and the rate can change later without breaking past comparisons.

Avoid sampling in these cases:

• Exact counts. Billing, quota enforcement, and SLA reporting need every event. Use metrics for these numbers, not sampled logs.
• Rare events. A failure that happens once an hour can disappear completely at 1 in 100.
• Compliance-constrained data. If a retention rule requires a complete record, a partial record can fail an audit.
• Teams that don't know the data is sampled. If analysts read raw counts as totals, sampling produces wrong conclusions. That is worse than paying for the full volume.

Copyimport hashlib

SAMPLE_RATE = 10  # keep 1 in 10 lines of the sampled class
EXEMPT_LEVELS = {"WARNING", "ERROR", "CRITICAL"}

def is_sampled_class(event: dict) -> bool:
    # Only high-volume, low-value lines are candidates: successful requests.
    return event.get("status") == 200 and event.get("level") == "INFO"

def keep_by_trace(trace_id: str, rate: int) -> bool:
    # Stable hash (not Python's per-process hash()) so every service
    # makes the same keep/drop decision for the same trace.
    digest = hashlib.sha256(trace_id.encode("utf-8")).digest()
    bucket = int.from_bytes(digest[:8], "big") % rate
    return bucket == 0

def filter_event(event: dict) -> dict | None:
    # Errors, warnings, and audit events skip the sampler entirely.
    if event.get("level") in EXEMPT_LEVELS or event.get("category") == "audit":
        return event
    if not is_sampled_class(event):
        return event
    trace_id = event.get("trace_id") or event.get("request_id")
    if trace_id is None:
        return event  # no key to group on; keep rather than sample randomly
    if keep_by_trace(trace_id, SAMPLE_RATE):
        # Record the rate so queries can scale counts back: sum(sample_rate).
        return {**event, "sample_rate": SAMPLE_RATE}
    return None  # dropped before ingest

if __name__ == "__main__":
    events = [
        {"level": "INFO", "status": 200, "trace_id": "4bf92f3577b34da6", "path": "/health"},
        {"level": "ERROR", "status": 500, "trace_id": "00f067aa0ba902b7", "path": "/orders"},
        {"level": "INFO", "category": "audit", "trace_id": "a3ce929d0e0e4736", "action": "role_change"},
    ]
    for e in events:
        print(filter_event(e))

A pre-ingest filter that samples successful request logs at 1 in 10 using a stable hash of the trace ID, passes errors, warnings, and audit events through unsampled, and stamps kept lines with a sample_rate field so counts can be scaled back.

What are the alternatives to filtering logs in the pipeline?

Filtering in the pipeline isn't the only way to cut volume, and it's often not the cheapest one to operate.

Convert repetitive events into metrics. A "request completed" line per request usually exists to answer questions about rate, latency and status codes. A counter plus a latency histogram answers those questions at a cost that scales with the number of series, not the number of requests. You lose per-event context such as request IDs and payload details, so keep logging errors and slow requests individually. Label choice decides the bill. A label like user ID or raw URL path creates one series per distinct value, and that high cardinality can cost more than the logs it replaced.

Fix noise at the source. Demote chatty messages to DEBUG, remove logging from hot loops, and rate-limit repeated identical messages in the application by logging the first occurrence and then a periodic suppressed-count summary. The noise is never serialized or shipped, so you also save CPU and network. The fix lasts because it doesn't depend on a central rule list staying current. The cost is code changes in every service, which means coordination across teams.

Most teams combine all three. Pipeline rules give immediate relief, while source fixes and metrics conversions shrink the rule list over time.

Copyimport logging
import time

class RepeatSuppressFilter(logging.Filter):
    """Log the first occurrence of a message, drop identical repeats for
    `window` seconds, then report how many were dropped."""

    def __init__(self, window=60.0):
        super().__init__()
        self.window = window
        self.state = {}  # key -> [window_start, suppressed_count]

    def filter(self, record):
        # Key on the unformatted template so "upstream %s timed out"
        # groups together across different argument values.
        key = (record.name, record.levelno, record.msg)
        now = time.monotonic()
        entry = self.state.get(key)
        if entry is None or now - entry[0] >= self.window:
            suppressed = entry[1] if entry else 0
            self.state[key] = [now, 0]
            if suppressed:
                # The count rides on the next real occurrence, not a separate line.
                record.msg = f"{record.getMessage()} [suppressed {suppressed} repeats since last report]"
                record.args = None
            return True
        entry[1] += 1  # Dropped here: never formatted, serialized or shipped.
        return False

# Attach to the handler so records propagated from child loggers are covered too.
handler = logging.StreamHandler()
handler.addFilter(RepeatSuppressFilter(window=30))
logging.basicConfig(level=logging.INFO, handlers=[handler])

log = logging.getLogger("payments")
for attempt in range(1000):
    log.debug("retry attempt %d", attempt)  # Hot-loop detail demoted to DEBUG.
    log.warning("upstream %s timed out", "ledger-api")  # Printed once per 30s window.

A source-side fix using Python's standard logging module: per-iteration detail is demoted to DEBUG, and a handler filter logs the first occurrence of a repeated message, drops repeats for 30 seconds, and appends the suppressed count to the next occurrence logged after the window.

How do you roll out log drop rules without losing the logs you need?

Start every new rule in dry-run mode. Instead of discarding matches, route them to a cheap destination or tag them and emit a count, then compare that count against the savings you estimated. If a rule meant to cut debug noise from one service matches 40% of all traffic, the condition is wrong. You want to find that out while the events still exist.

Roll out one service at a time, and start with the services whose owners will notice a gap. Dropping is irreversible. A log you discarded at ingest cannot be recovered during the incident where you needed it. A useful hedge is to send dropped events to compressed object storage with a short expiry. That gives you a recovery window at a fraction of indexing cost.

Alert on ingested volume per service against its own baseline, in both directions:

• A sudden drop often means a transform is failing silently. A condition that references a missing field can quietly match everything or nothing.
• A sudden spike means something new is escaping your rules.

Rules drift. A field renamed in a logging library refactor, or a reworded message, can turn a precise filter into a no-op or a blanket drop. Give each rule an owner, keep rules in version control next to the services they affect, and test them against sample events. Vector supports unit tests for transforms, and those tests catch format changes before deploy.

The approach is not worth it at low total volume. If your monthly log bill is smaller than the cost of an engineer's week spent building and maintaining rules, set sensible retention and stop there.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce logging costs, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.