Reduce Logging Costs: Sampling, Tiering and Dropping Noise at Ingest

Where logging money actually goes

A logging bill is the sum of several separate charges, and each one grows with a different thing. Most hosted and self-managed platforms charge in some form for five drivers: ingest volume, indexing, hot storage retention, query/scan and egress. Ingest and indexing are paid on every byte that arrives, so they usually make up most of the bill. Retention, by contrast, only applies to data you already paid to ingest. Shortening retention from 30 to 7 days cuts storage but leaves the per-GB ingest charge unchanged. You get more out of deciding, for each GB, whether it should be ingested at all.

That makes the pipeline the place to control the budget. A shipper or aggregator that sits in front of the backend can drop, sample or summarize events before they become billable.

This approach is aimed at backend and platform engineers who own services or run the shared logging pipeline. It fits when observability spend is high or growing, when a few chatty sources produce most of the volume, or when every log stream is kept for the same global retention period.

Finding the noise before cutting anything

Before writing any filter, measure where the bytes come from. Rank volume at three levels: per service, per logger (or source file), and per message template. Service-level totals show who to talk to. Logger-level totals show which component is responsible. Template-level totals show the exact rule you would need to write.

Raw message counts are misleading because every line is unique once it contains a request ID or a duration. To group similar lines, normalize each message by replacing variable parts with placeholders. Replace UUIDs and hex IDs with <id>, numbers with <n>, timestamps with <ts>, and IP addresses with <ip>. After normalization, GET /health 200 in 3ms req=8f2c… and GET /health 200 in 5ms req=a91d… both become GET /health <n> in <n>ms req=<id>.

You can then group by service and template, summing bytes rather than counting lines, and sort descending. A sample of a day's logs is enough for this. A few regex substitutions in sed, followed by sort | uniq -c | sort -rn, will do it. So will a GROUP BY template ORDER BY SUM(bytes) DESC in any SQL engine that can read your log export.

The result is usually lopsided. A small number of templates often account for a large share of volume. Typical culprits are:

• health-check and readiness probe lines
• debug logging left on inside loops
• one "request succeeded" line per call
• retry messages repeated on every attempt

This step pays for itself in engineering time. It narrows the work to a handful of targeted rules with known savings. It also avoids broad cuts, such as raising the log level on an entire service, which save less and can remove logs you will need during an incident.

Copy#!/usr/bin/env bash
# Rank message templates by total bytes, per service, from a one-day sample.
# Assumed line format: <ISO-8601 timestamp> <service> <message...>
# Usage: ./top-templates.sh sample.log
set -euo pipefail
LOG="$1"
export LC_ALL=C   # make awk length() count bytes, not characters

# Strip variable parts. Order matters: timestamps and UUIDs before plain numbers.
normalize() {
  sed -E \
    -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}[T ][0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?(Z|[+-][0-9]{2}:?[0-9]{2})?/<ts>/g' \
    -e 's/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}/<id>/g' \
    -e 's/[0-9a-fA-F]{8,}/<id>/g' \
    -e 's/[0-9]{1,3}(\.[0-9]{1,3}){3}/<ip>/g' \
    -e 's/[0-9]+/<n>/g' \
    "$1"
}

# Column 1: original line size in bytes (+1 for the newline).
# Column 2: normalized line. Rough pass: digits inside words or service
# names (e.g. "s3") also become <n>, which is fine for ranking.
paste -d '\t' <(awk '{ print length($0) + 1 }' "$LOG") <(normalize "$LOG") |
awk -F '\t' '{
  split($2, f, " \")
  svc = f[2]                              # second field is the service
  tmpl = $2
  sub(/^[^ ]+ [^ ]+ /, "", tmpl)          # drop <ts> and service, keep template
  bytes[svc "\t" tmpl] += $1              # sum bytes, not line counts
} END {
  for (k in bytes) printf "%d\t%s\n", bytes[k], k
}' |
sort -t "$(printf '\t')" -k1,1nr |
head -n 20   # the few templates worth writing rules for

A shell pipeline that normalizes log lines into templates with sed, sums their original byte size per service and template with awk, and lists the 20 largest so you can target the few rules worth writing.

How a cost-aware log pipeline works

Once the noisy log classes are known, the pipeline enforces decisions about them in a fixed order. Each stage passes on fewer events, so later stages do less work.

Copysources ──> parse/classify ──> filter (drop noise) ──> sample ──> route
 (apps, LB,     (structured        (health checks,      (by key)    │
  k8s, legacy)   fields, class)     debug spam)                     ├─> hot index      (errors, audit)
                      │                                             ├─> object storage (warm, sampled info)
                      └──> log_to_metric ──> metrics backend        └─> archive        (compliance)

Parsing comes first because every later rule depends on structured fields such as level, service and route. Regex over raw lines is fragile and slow. Logs-to-metrics branches off before sampling, so request counts and error rates are computed from every event, not from the sampled subset. Sampling should be deterministic. In Vector, sample with a key_field such as a trace ID keeps or drops whole requests together, which avoids keeping half of a request's lines. Routing comes last and sends each class to the cheapest tier that still fits how that class gets queried.

Source vs. pipeline drops

For services you own, fix the problem at the source. Remove chatty log lines and set sane default log levels. This saves serialization, network and agent CPU costs, not just storage. For third-party and legacy sources you can't change, use filter or route conditions in the pipeline.

Either way, keep drop rules in version control and send them through code review. A badly written condition can quietly delete the logs you will need in the next incident, so each change should show a diff, have a reviewer and be easy to revert.

Copysources:
  app: {type: file, include: ["/var/log/app/*.log"]}
transforms:
  parse:  # structured fields first; every later rule depends on them
    type: remap
    inputs: [app]
    source: '. = merge(., object!(parse_json!(string!(.message))))'
  request_metrics:  # branch off BEFORE sampling so counts see every event
    type: log_to_metric
    inputs: [parse]
    metrics:
      - {type: counter, field: route, name: http_requests_total, tags: {service: "{{ service }}", level: "{{ level }}"}}
  drop_noise:  # health checks and debug spam from sources we can't fix upstream
    type: filter
    inputs: [parse]
    condition: '.route != "/healthz" && .level != "debug"'
  sample_info:  # keep ~1 in 10 requests, whole trace together; never sample errors/audit
    type: sample
    inputs: [drop_noise]
    rate: 10
    key_field: trace_id
    exclude: '.level == "error" || .class == "audit"'
  by_class:  # send each class to the cheapest tier that fits how it is queried
    type: route
    inputs: [sample_info]
    route:
      hot: '.level == "error" || .class == "audit"'
      warm: '.level == "info"'
sinks:
  hot_index:
    type: elasticsearch
    inputs: [by_class.hot]
    endpoints: [""]
  warm_store:
    type: aws_s3
    inputs: [by_class.warm]
    bucket: example-logs-warm
    region: us-east-1
    encoding: {codec: json}
  metrics_out: {type: prometheus_exporter, inputs: [request_metrics]}

A minimal Vector pipeline that parses JSON logs and derives request metrics from every event. It then drops health-check and debug noise, applies trace-keyed sampling that never touches errors or audit logs, and routes errors to a hot index and sampled info logs to object storage.

Sampling and logs-to-metrics without losing the signal

Uniform sampling treats a checkout failure the same as a health-check success. That is the wrong default. Set rates per event class instead. Keep errors, warnings and security or audit events at 100%. Sample only the high-volume, low-information classes, such as successful reads, cache hits and routine polling.

Per-line random sampling has a quieter problem. A single request produces many lines across many services. If each line is sampled independently, you keep the authentication log but lose the database call that followed it, and you can no longer reconstruct the request. Key the decision on the trace or request ID: hash the ID, compare it against the threshold, and every line sharing that ID is kept or dropped together. This only works if all services propagate the same ID consistently. Where a service generates its own or drops the header, keyed sampling falls back to fragments.

Write the applied rate onto each kept event as a field, for example sample_rate: 0.1. Analysts can then recover true volume by summing 1 / sample_rate instead of counting rows. Without that field, dashboards silently undercount after every rate change.

For success paths you rarely read line by line, derive metrics in the pipeline before discarding the raw events. Emit a request counter and a latency histogram labelled by route, method and status class. Never promote user IDs, request IDs or full URLs to labels. Each unique value creates a new time series, and an unbounded label can cost more in your metrics backend than the logs you removed.

Copyimport hashlib

# Per-class rates for high-volume, low-information events only.
SAMPLE_RATES = {"http_success": 0.1, "cache_hit": 0.01, "health_check": 0.001}
ALWAYS_KEEP_LEVELS = {"ERROR", "WARN"}
ALWAYS_KEEP_CATEGORIES = {"security", "audit"}

def trace_bucket(trace_id: str) -> float:
    # Deterministic: every service must use the same hash so all lines of a
    # trace land in the same bucket in [0, 1).
    digest = hashlib.sha256(trace_id.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big") / 2**64

def sample(event: dict) -> bool:
    if (event.get("level") in ALWAYS_KEEP_LEVELS
            or event.get("category") in ALWAYS_KEEP_CATEGORIES):
        rate = 1.0
    else:
        rate = SAMPLE_RATES.get(event.get("event_class"), 1.0)

    trace_id = event.get("trace_id")
    # Without a propagated ID we cannot keep the request together; keep the
    # line rather than drop it silently, and fix propagation upstream.
    if rate >= 1.0 or not trace_id:
        event["sample_rate"] = 1.0
        return True

    # Same threshold comparison everywhere: traces kept at 1% are a subset of
    # traces kept at 10%, so lower-rate classes nest inside higher-rate ones.
    if trace_bucket(trace_id) < rate:
        event["sample_rate"] = rate  # lets queries re-weight counts
        return True
    return False

# Re-weighted count: sum 1/sample_rate instead of counting rows.
# SELECT route, SUM(1.0 / sample_rate) AS estimated_requests
# FROM logs
# WHERE event_class = 'http_success'
# GROUP BY route;

Trace-keyed, per-class sampling that always keeps errors, warnings and audit/security events, writes the applied sample_rate onto each kept event, and recovers true volume with a SUM(1/sample_rate) query.

Designing log retention tiers by query pattern

Retention is the other half of the bill. Most log queries hit the last few days: incident triage, deploy verification, a customer ticket from this morning. Paying for indexed, replicated storage on month-old debug output buys speed nobody uses.

A tiered layout matches storage to how each log class is actually queried:

Decide routing per log class at ingest, not with one global retention period. A single global number is always set by the strictest requirement, usually audit, so every chatty info log ends up kept as long as compliance data. Tagging class at the pipeline stage also lets audit logs skip the hot index entirely if nobody searches them interactively.

The alternatives are worth weighing:

• Shorter global retention: simplest option, and it works if no class needs long retention. It breaks as soon as one does.
• Index-free or schema-on-read store: cheap ingest because nothing is parsed or indexed up front. Each query pays for that in scan time and cost.
• Plain object storage with on-demand query: lowest steady-state cost. It fits teams that rarely query old data and can accept slow, occasionally expensive searches when they do.

Illustrative estimate: how much each lever changes the bill

Illustrative estimate (assumed numbers, not a benchmark). Assume 500 GB/day (15,000 GB/month) and $0.50/GB for ingest and indexing. Assume $0.10/GB-month for hot storage and $0.01/GB-month for archive, with 30-day retention. Assume 30% of volume is noise, and 40% of what remains is high-volume success logs sampled at 10%.

Monthly cost = (GB ingested × ingest price) + (GB in hot × hot price) + (GB in archive × archive price). At steady state, the stored GB equal monthly volume × days retained ÷ 30.

Order matters because ingest is charged before any tiering decision is made. If you tier the raw stream without dropping first, the bill is about $7,965, because every noisy gigabyte is still indexed at $0.50. Shortening retention by itself trims only the storage line. The ingest line accounts for 83% of this baseline, and retention does not touch it.

On the engineering side, assume three engineer-weeks to build the pipeline plus a few hours each month to review the filters and sample rates. Compare that with about $5,400/month in recurring savings, and redo the comparison using your own loaded engineering cost. The case gets weaker if your noise share or your ingest price is much lower than these assumptions.

When to avoid aggressive cuts, and how to keep savings from drifting

Savings only count if they don't cost you an incident. Before tightening any rule, check it against the failure modes below.

The pipeline is also a production system in its own right. It consumes CPU and needs on-call ownership. Under back-pressure it can shed events without anyone noticing. Give it bounded buffers, preferably disk-backed for critical sources, and alert when those buffers fill.

Guardrails

• Per-source volume alerts. These catch a service whose output suddenly multiplies before the invoice does.
• Rate limits or quotas. These stop one runaway service from flooding shared ingest.
• Per-rule counters of dropped and sampled events. During an incident, an engineer can see in seconds whether a missing log line was filtered or never emitted. That saves hours of guessing.

Making it stick

• Team ownership. Assign each team ownership of its own log volume.
• Monthly cost-per-service reports. These make spend visible to the people who generate it.
• Code review for log changes. Review new high-volume log statements and new drop rules the same way you review schema changes.

Without these habits, the new noise from each release accumulates quietly, and the bill drifts back toward where it started within a few quarters.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce logging costs, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.