How to Reduce Log Ingestion Cost: Sampling, Levels and Retention

Audit first: find which sources and levels drive your observability bill

Log ingestion cost is what you pay to ship, parse, index and store each log event. Most vendors bill on volume (ingested gigabytes, sometimes plus indexed or retained volume), so the unit to measure is bytes, not lines. A single 8 KB stack trace can cost as much as hundreds of short access-log lines. Ranking by line count will point you at the wrong targets.

Volume often concentrates in a few sources: chatty health checks, debug logging left on in production, retry loops, and full request or response bodies. Don't assume this pattern holds for you. Measure it, because your top contributors may be somewhere you didn't expect.

What to break down

Group by message template (the format string with variables stripped out) to find the repeated line you can fix once. Sort by total bytes over a representative window, such as a full week, so daily batch jobs show up. Record the baseline before you change anything, or you won't be able to prove the savings.

The audit belongs to the backend and platform engineers who own the services and the collector pipeline. Run it in three situations:

• Before any cuts
• After an unexplained bill spike
• As part of a periodic review, so that new sources don't creep back in unnoticed

Copy#!/usr/bin/env python3
"""Rank log volume by BYTES, grouped by service, level and message template.
Usage: python3 -I log_audit.py week-of-logs/*.jsonl   (one JSON event per line)
Run it on a representative window (e.g. 7 days) and save the output as your baseline."""
import fileinput, json, re
from collections import Counter

# Strip variable parts (UUIDs, hex ids, numbers) so lines that differ only by
# parameters collapse into one template you can fix once in code.
VARIABLE = re.compile(r"[0-9a-f]{8}-[0-9a-f-]{27}|0x[0-9a-f]+|\d+", re.I)

def template(msg):
    return VARIABLE.sub("<*>", msg)[:100]

bytes_by_key, lines_by_key, total = Counter(), Counter(), 0
for raw in fileinput.input(mode="rb"):
    size = len(raw)  # measure raw bytes as shipped, not line count
    total += size
    try:
        event = json.loads(raw)
    except ValueError:
        event = None
    if not isinstance(event, dict):
        event = {"service": "unparsed", "message": "<non-JSON line>"}
    key = (str(event.get("service", "unknown")),
           str(event.get("level", "unknown")).upper(),
           template(str(event.get("message", ""))))
    bytes_by_key[key] += size
    lines_by_key[key] += 1

# Top contributors by total bytes; avg size exposes stack traces and bodies.
print(f"{'share':>6} {'MB':>9} {'lines':>10} {'avg B':>7}  service | level | template")
for key, b in bytes_by_key.most_common(20):
    n = lines_by_key[key]
    print(f"{b / max(total, 1):6.1%} {b / 1e6:9.1f} {n:10d} {b // n:7d}  {' | '.join(key)}")
print(f"total: {total / 1e9:.2f} GB across {sum(lines_by_key.values())} lines")

A standard-library Python script that reads JSON-lines logs, sums raw bytes per service, level and message template, and prints the top 20 contributors by byte share so you can record a baseline before cutting anything.

Fix log levels and remove waste at the source

The cheapest log line is the one that is never emitted. Fix the source before you tune any pipeline, because sampling and filtering downstream still cost serialization, network and collector CPU.

• Set the production default to INFO or WARN. DEBUG belongs in development. Most services settle on INFO. Pick WARN for high-throughput services where INFO is mostly noise.
• Demote success-path messages to DEBUG. Lines such as "request received", "cache hit" and "handler completed" fire on every request and rarely help diagnose anything. Keep INFO for state changes and decisions an on-call engineer would need.
• Make levels adjustable at runtime, per logger. Expose a control (an admin endpoint, a config watch or a feature flag) that changes the level for one package or module without a restart. Engineers can then raise verbosity on the suspect component during an incident and leave the rest of the fleet quiet.
• Strip waste before sampling:
  • Duplicate fields, such as the same trace ID or hostname added by both the app and the collector
  • Oversized payloads. Truncate or cap field length.
  • Full request and response bodies. Log identifiers and sizes instead.
  • Stack traces repeated on every retry. Log the trace once and count the retries.
  • Data that metrics or traces already capture, such as per-request latency

Trade-off: A lower default level moves cost to incident time. If you lack runtime adjustment, the usual fix is an emergency deploy with DEBUG enabled everywhere. That deploy can spike the bill within hours and slow the investigation it was meant to help.

Copyimport logging
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs

# Production default: INFO. DEBUG lines are dropped before they are formatted or shipped.
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
log = logging.getLogger("checkout.payments")

def handle(req_id: str, body: bytes, declined: bool) -> None:
    log.debug("handler completed req_id=%s", req_id)  # success path: DEBUG only
    if declined:  # a decision on-call needs: keep at INFO, log size not the body
        log.info("payment declined req_id=%s body_bytes=%d", req_id, len(body))

def call_with_retry(fn, attempts: int = 3):
    for i in range(1, attempts + 1):
        try:
            return fn()
        except Exception:
            if i == 1:
                log.exception("upstream call failed, retrying")  # stack trace once
            if i == attempts:
                log.error("upstream call failed after %d attempts", attempts)  # count, no trace
                raise

LEVELS = {"DEBUG": logging.DEBUG, "INFO": logging.INFO, "WARNING": logging.WARNING}

class LogLevelAdmin(BaseHTTPRequestHandler):
    # PUT /loglevel?logger=checkout.payments&level=DEBUG raises one logger, no restart
    def do_PUT(self):
        q = parse_qs(urlparse(self.path).query)
        name, level = q.get("logger", [""])[0], q.get("level", [""])[0].upper()
        if not name or level not in LEVELS:
            self.send_response(400); self.end_headers(); return
        logging.getLogger(name).setLevel(LEVELS[level])
        self.send_response(204); self.end_headers()

if __name__ == "__main__":
    # Bind to localhost only; expose through your normal authenticated admin path.
    HTTPServer(("127.0.0.1", 9091), LogLevelAdmin).serve_forever()

A Python standard-library example that defaults logging to INFO, demotes success-path lines to DEBUG, logs a retry stack trace only once, and adds a local admin endpoint that changes one logger's level at runtime without a redeploy.

How log sampling works without losing the events that matter

Uniform sampling, such as keeping 10% of everything, is unsafe because cost and value are distributed unevenly. Most of your volume comes from repetitive success paths, while most of your diagnostic value sits in rare lines. A flat rate drops nine in ten errors along with nine in ten health checks. Sample selectively instead:

• Keep 100% of errors, warnings, security and audit events. They are low-volume, and losing one can mean losing the only evidence of an incident or a compliance gap.
• Sample high-volume success paths at a fixed rate, for example 1 in 20 for routine 200 request logs.
• Rate-limit or deduplicate log storms per key or message template. When a retry loop emits the same line thousands of times a second, keep the first N per window and then emit a summary count. Rate-limiting this way keeps the storm visible without paying for every copy.
• Record the sample rate on each event, as a field like sample_rate=20. Queries can then re-weight counts, and nobody mistakes sampled volume for real traffic.
• Make sampling trace-aware. Derive the keep or drop decision from the trace ID, for instance by hashing it against a threshold, so every service makes the same choice. A sampled request then keeps all of its log lines end to end, instead of a scattered set of fragments from different requests.

Avoid sampling in two cases. The first is rare events nobody anticipated, because no rule protects them and sampling quietly discards them. The second is count-style questions such as "how many checkouts failed?" A metric answers these exactly, while re-weighted sampled logs only give an approximation.

Copyimport hashlib, logging, random, time

SUCCESS_RATE = 20   # keep 1 in 20 routine success lines
STORM_LIMIT = 100   # max lines per message template per window
WINDOW_S = 60
summary_log = logging.getLogger("log_sampler")  # separate logger, not filtered

def trace_kept(trace_id, rate):
    # Same trace ID -> same decision in every service, so sampled requests stay whole
    bucket = int(hashlib.sha256(trace_id.encode()).hexdigest()[:8], 16)
    return bucket % rate == 0

class SelectiveSampler(logging.Filter):
    def __init__(self):
        super().__init__()
        self.window_start, self.counts = time.monotonic(), {}

    def filter(self, record):
        record.sample_rate = 1
        # Never sample warnings, errors, security or audit events
        if record.levelno >= logging.WARNING or getattr(record, "category", "") in ("security", "audit"):
            return True
        # Storm control keyed on the unformatted template, e.g. "retry %s failed"
        now = time.monotonic()
        if now - self.window_start >= WINDOW_S:
            for tpl, n in self.counts.items():
                if n > STORM_LIMIT:
                    summary_log.info("suppressed %d copies of %r", n - STORM_LIMIT, tpl)
            self.window_start, self.counts = now, {}
        self.counts[record.msg] = self.counts.get(record.msg, 0) + 1
        if self.counts[record.msg] > STORM_LIMIT:
            return False
        # Trace-aware sampling for high-volume success paths; random only if no trace ID
        trace_id = getattr(record, "trace_id", None)
        kept = trace_kept(trace_id, SUCCESS_RATE) if trace_id else random.randrange(SUCCESS_RATE) == 0
        record.sample_rate = SUCCESS_RATE  # lets queries re-weight counts
        return kept

logging.getLogger("app").addFilter(SelectiveSampler())
# usage: log.info("GET /cart 200", extra={"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"})

A standard-library Python logging filter that keeps all warnings, errors, security and audit events, caps repeated message templates per window and emits a suppressed-count summary, samples success lines by hashing the trace ID, and records sample_rate on every event.

Where to cut: application, collector pipeline or vendor layer

The stage where you drop a log line decides whether you pay for it. Most vendors charge on bytes received, so anything discarded before it leaves your network is never billed.

Copy app ──[A]──▶ collector/agent ──[B]──▶ vendor ingest ──[C]──▶ index
                                                               │
                                              hot ──▶ warm ──▶ archive
 [A] level gates, in-process sampling     → never sent, never billed
 [B] filter, sample, drop fields          → never sent, never billed
 [C] exclusion rules, index-time drops    → often billed as ingest anyway

Cuts at A are the cheapest because the line is never even serialized. Cuts at B let you apply one policy across many services without redeploying them. Cuts at C often reduce only storage or indexing charges. The bytes have already crossed the ingest meter. Read your provider's pricing page for the stages it bills: ingest, indexing, retained volume, and scanned-at-query. Two pipelines with identical drop rules can produce very different bills depending on where those rules run.

Trade-offs of pipeline-side filtering

• Resource cost. Parsing and matching every line takes CPU and memory. That cost sits on your nodes or on a dedicated collector tier, and it grows with volume.
• Bottleneck and failure point. A central collector that falls behind applies backpressure or drops data. If it goes down, logs stop flowing. Size buffers deliberately and run more than one replica.
• Silent drops. A rule that matches a message string or field name keeps matching after a schema change, so it can start discarding the wrong events. A rule can also stop matching and let unwanted volume through. Neither raises an error. Emit a counter of dropped lines per rule and alert when it changes sharply.

Copy# OpenTelemetry Collector (contrib distribution): cut point [B].
# Everything dropped here is never sent, so it never crosses the vendor's ingest meter.
receivers:
  otlp:
    protocols:
      grpc:

processors:
  # Guard the collector itself: refuse data before it runs out of memory.
  memory_limiter:
    check_interval: 1s
    limit_mib: 1500
    spike_limit_mib: 300

  # Drop DEBUG/TRACE records and health-check noise.
  # These rules match on severity and message text. After a schema or log-format
  # change, re-check them, because a stale match fails silently.
  filter/drop_noise:
    error_mode: ignore
    logs:
      log_record:
        - 'severity_number < SEVERITY_NUMBER_INFO'
        - 'IsMatch(body, "GET /healthz")'

  # Strip high-volume or sensitive fields you never query.
  attributes/trim:
    actions:
      - key: http.request.header.cookie
        action: delete

  batch:

exporters:
  otlphttp:
    endpoint:

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, filter/drop_noise, attributes/trim, batch]
      exporters: [otlphttp]

An OpenTelemetry Collector logs pipeline drops low-severity and health-check records and strips unused fields before export, which is cut point B in the diagram, with a memory limiter placed first to keep the collector from becoming the failure point.

Define a log retention policy by log class

Retention cost grows with every day a line stays in indexed storage, so the default should not be one window for everything. Set retention by log class, based on who queries that class, how often, and how fast they need answers.

Debug logs lose value quickly. Few people search last quarter's stack traces, so keep them in the expensive searchable tier only as long as your incident reviews actually reach back. Data that must be kept but is rarely read can sit in compressed object storage, which costs far less per gigabyte than indexed storage.

The cheap tiers have costs of their own. Archived data is slow to query, often taking minutes to hours rather than seconds. Rehydrating it can also add retrieval or query charges, so one large investigation can wipe out months of storage savings.

The bigger risk is security. Breach investigations often need logs from well before the debugging window, because intrusions tend to be found long after they start. Audit log retention is a legal decision, not an engineering preference. Before you shorten any retention period, get sign-off from security and legal, and record who approved it.

Alternatives: log-derived metrics, traces and blanket vendor-side cuts

Dropping lines is not the only way to shrink volume. Three alternatives come up often, and each moves cost somewhere rather than making it disappear.

Log-derived metrics. For a high-volume, low-information pattern such as a health-check hit or a cache miss, the collector can count matching lines and emit a counter instead of forwarding each one. This works when you only need rates. You lose per-event context, so you can no longer pull up the individual request behind a spike. Watch the labels, too. Extracting user_id or a raw URL path as a label creates a new time series per value. The cost then lands on your metrics backend, and many backends charge by active series.

Traces. If logs mostly exist to reconstruct a request's path, tracing with tail-based sampling can carry that detail. That only works if instrumentation coverage is good. Sampled-out requests leave nothing behind.

Blanket vendor-side cuts. Dropping or downgrading data after it reaches the vendor is fast to configure. Under ingest-based pricing, though, you may already have paid for those bytes.

Pricing models differ: some providers bill on ingest, some on queries, some per host. A tuning that saves money under one model may not carry over to another. Cutting volume at the source is the saving that stays with you if you switch providers.

Impact on cost and engineering time, with guardrails so savings last

Illustrative estimate (assumed numbers, not a benchmark)

Every figure below is an input chosen for the example, not a measurement. Assume 500 GB/day of logs, a $0.50/GB ingest price, $0.10/GB-month for hot storage, DEBUG at 40% of volume, and half of the remaining volume being routine successful-request INFO logs.

Monthly cost = (GB/day × 30 × ingest price) + (GB/day × hot retention days × storage price)

The levers stack because each one shrinks what the next one applies to. Once volume falls, retention changes save less in absolute terms.

Weigh the engineering cost against the savings. Changing log levels, adding sampling rules and writing retention policies takes developer time, and it competes with feature work. You get time back in other ways. Searches run faster over smaller indexes. On-call engineers read less noise during incidents. Fewer surprise invoices mean fewer emergency volume cuts made under pressure.

Guardrails so savings last

• Set a volume budget for each service, or alert on ingest growth week over week, so a regression shows up within days and not on the next invoice.
• Flag new logging in hot paths (request loops, retries, per-item processing) during code review.
• Attribute cost to each team so the owners of a regression can see it and fix it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce log ingestion cost, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.