Reduce Logging Costs for APIs: Sampling, Levels and Retention Tiers

What it is

Reducing logging costs for an API means controlling how many log events you produce, how large each one is, how many you keep, and how long and where you store them, while preserving the events you need to debug incidents, answer audit questions and drive dashboards. It is not one tool. It is a set of policies applied at different points:

• Log levels decide what the application emits at all.
• Sampling decides which emitted events are kept.
• Event trimming and rate limiting decide how many bytes each kept event costs.
• Retention tiers decide how long data stays fully indexed and where it moves afterward.
• Metrics take over the counting work that sampled logs can no longer do accurately.

Who should use it

This approach is for teams whose log bill or log volume grows faster than their traffic, or whose incident searches return too much noise to be useful. In practice that means:

• Backend engineers who own what each service logs and at which level.
• Platform and SRE engineers who run the collection pipeline, set sampling and routing rules, and manage the vendor contract and retention policy.
• Security and compliance owners, who need to sign off on what may be sampled, trimmed or archived.

It is less useful for small services with low, steady log volume. There, the engineering time spent on sampling and tiering can cost more than the logs themselves. Logs that must be kept in full for legal or security reasons are also out of scope for sampling, though they still benefit from tiered storage.

How it works

Each event passes through a series of decisions, and each decision reduces a different part of the bill:

1. At the source, the configured log level filters out events below the threshold. Runtime overrides with an expiry let you raise detail for one module during an investigation.
2. In the application or at ingress, a sampling function always keeps errors, slow requests and security events. It keeps a deterministic fraction of successful requests, chosen by hashing the trace ID, and writes the applied sample_rate onto each kept event.
3. Before emission, events are trimmed. Bodies are replaced by their size, fields the collector already adds are dropped, and repetitive messages are rate-limited with a summary count.
4. In parallel, metrics middleware records exact request counts and latency histograms for every request, so dashboards do not depend on sampled logs.
5. In a collector you run, tail rules make outcome-based keep/drop decisions. Routing rules then send each event to a hot, warm or archive tier before it reaches the vendor's ingest point.

The sections below cover each step in that order, starting with measuring where the volume comes from.

Inventory log volume and map it to what you are billed for

To reduce logging costs without losing the signal you rely on during incidents, treat log volume as a budget to allocate, not a number to cut across the board. A flat percentage reduction applied to every service removes useful context and leaves the waste in place. Backend engineers decide what each service emits. Platform engineers own the pipeline, the vendor contract and the retention policy. Both groups need the same inventory before changing anything.

Find the few event types that dominate

Group a recent window of logs by service, route, level and message template. A message template is the log string with variable values stripped out, so user 123 not found and user 456 not found count as one event type. Rank the groups by bytes, not line count, because one serialized payload can outweigh hundreds of short lines.

The distribution is often uneven, with a small number of groups accounting for a large share of the bytes. Your own inventory will show whether that holds for your services. Common candidates include:

• Load balancer health checks writing access logs every few seconds per instance.
• Debug logging enabled during an incident and never turned off.
• Full request or response bodies serialized into a single field.

Map each lever to a line item

Observability spend is often split into separate charges for ingestion, indexing, storage duration and query or scan volume. Check your own pricing to see which charges apply. Each lever reduces a different charge:

Set log levels by purpose, with runtime toggles that expire

Levels only reduce cost if each one has a clear purpose. Without that, everything drifts toward INFO.

• ERROR: a request or job failed and someone may need to act. If nobody would ever act on it, it is not an error.
• WARN: something degraded but recovered, such as a retry that succeeded or a fallback that was used. These are worth counting, and often belong in a metric rather than a log line.
• INFO: business-meaningful state changes, such as an order placed, a payment captured or a config reloaded. Routine events like "handler entered" or "query started" do not belong here.
• DEBUG: off by default in production. It carries the detail you want during an investigation and nowhere else.

To make lower defaults workable, levels need to change at runtime without a redeploy. They should be scoped per service and per module or logger, so you can raise one noisy component without flooding ingestion from the rest of the fleet. Every override should carry an expiry time and revert automatically. Without expiry, a debug flag set during Tuesday's incident is still running three weeks later and quietly undoes the savings. Record who set each override and why, so expired or abandoned ones are easy to audit.

The trade-off

Quieter defaults save money every day until an incident happens. Then the first occurrence of a failure has only ERROR and INFO context. A runtime toggle helps only with problems that recur after you enable it. A one-off failure is gone before you flip the switch. That gap is why the sampling and tail-retention decisions in the next section matter: they keep some full-fidelity context without paying for DEBUG everywhere.

Copyimport logging, threading, time

MAX_TTL_SECONDS = 4 * 3600  # hard cap: no debug session outlives a shift
_overrides = {}             # logger name -> override record, for auditing
_lock = threading.Lock()
audit = logging.getLogger("ops.log_overrides")

def set_level_override(name, level, ttl_seconds, set_by, reason):
    """Raise or lower one logger (module) temporarily; it always reverts."""
    ttl = min(ttl_seconds, MAX_TTL_SECONDS)
    logger = logging.getLogger(name)
    with _lock:
        prev = _overrides.pop(name, None)
        if prev:
            prev["timer"].cancel()  # extending keeps the ORIGINAL level to revert to
        original = prev["original"] if prev else logger.level  # 0 = NOTSET, inherit
        timer = threading.Timer(ttl, _revert, args=(name,))
        timer.daemon = True
        _overrides[name] = {"original": original, "expires_at": time.time() + ttl,
                            "set_by": set_by, "reason": reason, "timer": timer}
        logger.setLevel(level)
        timer.start()
    audit.warning("override %s=%s ttl=%ss by=%s reason=%s",
                  name, logging.getLevelName(level), ttl, set_by, reason)

def _revert(name):
    with _lock:
        entry = _overrides.pop(name, None)
        if entry:
            logging.getLogger(name).setLevel(entry["original"])
    if entry:
        audit.warning("override %s expired, level restored", name)

logging.basicConfig(level=logging.INFO)

set_level_override("orders_api.payments_client", logging.DEBUG,
                   ttl_seconds=1800, set_by="oncall@example.com",
                   reason="INC-1234 intermittent capture timeouts")

A per-process sketch of scoped, self-expiring log level overrides in Python: one module is raised to DEBUG with a capped TTL, the override records who set it and why, and the original level is restored automatically.

How log sampling works: layered head and tail decisions

Sampling works best as layers, each answering a different question about the event.

Layer 1: always keep. Errors, requests slower than your latency objective, and security events (auth failures, permission denials, rate-limit trips) bypass sampling entirely. These are rare and expensive to lose, so their cost is small relative to their value.

Layer 2: deterministic head sampling. For successful requests, decide at ingress by hashing the trace or request ID into a number between 0 and 1 and keeping the event if it falls below the configured rate. Because the decision depends only on the ID, every service that sees the same ID makes the same choice, so you keep or drop whole requests rather than scattered fragments. The example below does this and writes the rate onto the event (for instance sample_rate: 0.05), which lets downstream queries reweight counts.

Layer 3: tail decisions in a collector. Some keep/drop choices depend on the outcome, such as whether any span in the trace errored. A collector buffers all events for a trace until it completes, then applies the policy.

Where to hold back:

• A successful but wrong request, such as a 200 with a bad price, looks like every other success and is usually dropped. Keep sampling off for endpoints where correctness bugs are costly.
• Tail sampling costs collector memory proportional to traffic multiplied by the buffer window. It also delays delivery and adds a stateful component you must scale and operate.
• If services hash different keys or propagate IDs inconsistently, head decisions diverge and traces fall apart. Standardize the key before you lower any rate.

Copyimport hashlib

SUCCESS_SAMPLE_RATE = 0.05  # keep 5% of ordinary successful requests
SLOW_MS = 500               # latency objective; slower requests are always kept
SECURITY_EVENTS = {"auth_failure", "permission_denied", "rate_limited"}


def hash_to_unit(key: str) -> float:
    # Map an ID to [0, 1). Uses SHA-256 rather than Python's built-in hash(),
    # which is randomized per process, so every service gets the same answer.
    digest = hashlib.sha256(key.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big") / 2**64


def should_keep(event: dict) -> bool:
    # Layer 1: errors, slow requests and security events bypass sampling.
    if (
        event.get("status", 200) >= 500
        or event.get("duration_ms", 0) > SLOW_MS
        or event.get("event_type") in SECURITY_EVENTS
    ):
        event["sample_rate"] = 1.0
        return True

    # Layer 2: deterministic head sampling keyed on the propagated trace ID,
    # so all services keep or drop the same whole request.
    if hash_to_unit(event["trace_id"]) < SUCCESS_SAMPLE_RATE:
        # Record the rate so queries can reweight: estimated count = 1 / rate.
        event["sample_rate"] = SUCCESS_SAMPLE_RATE
        return True
    return False


event = {"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "status": 200,
         "duration_ms": 42, "route": "/v1/orders"}
if should_keep(event):
    print(event)  # hand off to your log shipper here

A deterministic keep/drop function that always keeps errors, slow requests and security events, head-samples successful requests by hashing the trace ID, and writes the applied sample_rate onto each kept event for later reweighting.

Keep dashboards accurate after sampling: metrics and reweighting

Once you sample, any dashboard that counts log lines starts lying. If a service keeps 1 in 20 successful requests, a panel showing 5,000 requests per minute really reflects about 100,000. The fix is to write the sample rate onto every retained event, for example as a sample_rate field holding 20. A query can then sum that field instead of counting rows, and each kept event stands in for the ones that were dropped. This only works if the field is always present and always accurate. When a tail rule keeps every error at rate 1 and samples successes at rate 50, a plain count of the mixed stream over-represents failures by a wide margin. Reweighting is what makes the numbers usable again.

Reweighted counts are still estimates, and they get noisy for low-traffic endpoints. Request rate, error rate and latency percentiles should come from metrics:

• Counters and histograms are updated on every request before any sampling decision, so they are exact and cheap to store no matter the traffic.
• Instrumentation is real work. You need middleware that records the route template rather than the raw path, status class and duration. You also need to cap label cardinality so metrics do not become the new cost problem.
• Expect pushback. Engineers who are used to slicing raw logs by any field will lose some ad hoc queries. Agree on which questions metrics must answer, and keep unsampled errors and traces for the long tail.

Copyimport json
import logging
import random

from prometheus_client import Counter, Histogram

REQUESTS = Counter("http_requests_total", "Requests handled", ["route", "status_class"])
LATENCY = Histogram("http_request_duration_seconds", "Request latency", ["route"])

SUCCESS_SAMPLE_RATE = 50  # keep roughly 1 in 50 successful requests
log = logging.getLogger("access")


def record_request(route_template, status, duration_s, extra_fields):
    status_class = f"{status // 100}xx"

    # 1. Metrics are updated on every request, before any sampling decision,
    #    so request rate, error rate and latency percentiles stay exact.
    REQUESTS.labels(route=route_template, status_class=status_class).inc()
    LATENCY.labels(route=route_template).observe(duration_s)

    # 2. Logs are sampled: server errors are always kept, successes at 1/50.
    rate = 1 if status >= 500 else SUCCESS_SAMPLE_RATE
    if random.randrange(rate) != 0:
        return

    # 3. Every retained event carries the rate it was kept at, so a query can
    #    use SUM(sample_rate) instead of COUNT(*) to estimate the true volume.
    event = {
        "route": route_template,
        "status": status,
        "duration_ms": round(duration_s * 1000, 1),
        "sample_rate": rate,
        **extra_fields,
    }
    log.info(json.dumps(event))

Request middleware logic in Python that updates Prometheus counters and histograms on every request, then samples log events and records each event's sample_rate so log-based counts can be reweighted.

Cut bytes per event at the source

Sampling reduces how many events you keep. The other lever is how large each kept event is.

• Drop or truncate bodies. Request and response payloads are usually the largest fields, and they often contain personal data. Log the size and content type instead. If you need a body for debugging, cap it at a fixed length and only capture it on error paths.
• Remove fields the pipeline already adds. If your collector attaches host, pod, region and service version, writing them from the application doubles the bytes. Also drop timestamps formatted twice and repeated static labels.
• Skip full stack traces for known errors. A database timeout that fires thousands of times an hour does not need a 60-line trace on every event. Log a stable error code and fingerprint on each occurrence, and the full trace once per process or once per interval.
• Rate-limit noisy messages. Key a counter on message template plus error code. Emit the first N events per window, suppress the rest, and when the window closes, write one summary event with the suppressed count. Totals stay correct and the volume stays bounded.

As an illustrative estimate, assume an event of several kilobytes that carries an embedded payload, duplicated host metadata and a full trace. Trimmed as in the example below, it could shrink to a few hundred bytes. What remains is the route, status, latency, error code and identifiers. The actual reduction depends on your payload sizes and field set, so measure it on your own events.

The risk is losing context during an incident. Never trim trace_id, request_id, tenant or user identifiers, or the error fingerprint. With those fields you can still join a trimmed log to traces, metrics and the occasional fully captured event.

Copyimport json, time
from collections import defaultdict

def slim_event(route, status, latency_ms, error_code, fingerprint,
               trace_id, request_id, tenant_id, body_bytes):
    return {"route": route, "status": status, "latency_ms": latency_ms,
            "error_code": error_code, "error_fingerprint": fingerprint,
            "trace_id": trace_id, "request_id": request_id,  # never trim: needed to correlate
            "tenant_id": tenant_id, "req_body_bytes": body_bytes}

class RateLimiter:
    """Emit the first N events per (template, error_code) per window, count the rest."""
    def __init__(self, max_per_window=20, window_s=60):
        self.max, self.window = max_per_window, window_s
        self.state = defaultdict(lambda: {"start": 0.0, "count": 0, "suppressed": 0})

    def log(self, template, event):
        key = (template, event.get("error_code"))
        s, now = self.state[key], time.monotonic()
        if now - s["start"] >= self.window:
            # Window closed: one summary event keeps totals correct.
            # (Flushed lazily on the next event for this key; a timer could flush idle keys.)
            if s["suppressed"]:
                print(json.dumps({"msg": "suppressed_summary", "template": template,
                                  "error_code": key[1], "suppressed": s["suppressed"],
                                  "window_s": self.window}))
            s.update(start=now, count=0, suppressed=0)
        s["count"] += 1
        if s["count"] <= self.max:
            print(json.dumps({"msg": template, **event}))
        else:
            s["suppressed"] += 1  # counted, not written

limiter = RateLimiter(max_per_window=20, window_s=60)
limiter.log("db query timed out", slim_event(
    "/v1/orders", 504, 3012, "DB_TIMEOUT", "a1f3c9e2",
    "4bf92f3577b34da6a3ce929d0e0e4736", "req-7f2c19", "tenant-042", 1834))

A trimmed structured log event that keeps correlation IDs, plus a per-template rate limiter that writes the first N events per window and a summary with the suppressed count.

Design retention tiers and route them in a central pipeline

Not every log line needs to be stored the same way. Assign retention by how each log is queried. If your incident queries mostly look at recent data, which you can confirm from your query history, only that recent window needs full indexing.

Make the routing decision in a collector or agent that you run, before the vendor's ingest point:

Copyservices ──> collector (filter, sample, route) ──┬──> hot
                                                 ├──> warm
                                                 └──> archive

When filtering, sampling and tier rules live in one place, you change them with a config push and don't need to redeploy services. Data you drop or archive before ingest also never reaches the ingest bill.

When not to tier aggressively

• Archives are slow and costly to scan. If a team regularly queries data older than the hot window, rehydration fees can cancel out the savings. Extend the warm tier for those logs.
• Audit and security logs follow separate rules. Legal holds, tamper evidence and forensic chain of custody set their retention, not cost. Route them to a dedicated, access-controlled sink, and never apply sampling to them.
• The pipeline becomes a critical component. A bad rule can silently drop error logs from every service. Review collector config changes like code, test the rules against sample events, and alert when the volume for a tier drops unexpectedly.

Impact on cost and engineering time, plus the rollout checklist

Illustrative estimate (assumed numbers, not a benchmark)

Formula: daily GB = req/s × 86,400 × events per request × kept fraction × KB per event ÷ 10⁶. Stored volume = daily GB × retention days.

Assumed: 1,000 req/s, 3 events per request, 1 KB per event, 1% error rate.

• Baseline: 1,000 × 86,400 × 3 × 1 × 1 KB ≈ 259 GB/day, or about 7.8 TB at 30 days.
• Drop debug in production (3 → 2 events): ≈ 173 GB/day.
• Keep all errors and 10% of successes (kept fraction 0.109): ≈ 19 GB/day.
• Trim events to 0.6 KB: ≈ 11 GB/day. With 7 days hot, that is about 79 GB searchable, and the rest goes to cheap archive.

Engineering time

Smaller indexes mean fewer irrelevant hits when you search during an incident. When sampling and routing live in the central pipeline, a policy change is one reviewed config change instead of a redeploy of every service. The cost comes up front. Teams need to add consistent fields, error flags and trace IDs before sampling can safely keep the right events.

Rollout checklist

• Each log category has a named owning team.
• Each service has a volume budget with spike alerts. Errors and security events are exempt from sampling.
• Legal and security have reviewed retention periods.
• Incident runbooks have been replayed against sampled data and still lead to the root cause.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to reduce logging costs, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.