Backend Monitoring Basics: Metrics, Logs, Traces and SLOs

What metrics, logs and traces each answer

Backend monitoring is the practice of collecting signals from the services you run so you can tell, quickly and with evidence, whether users are being served well. It falls to backend and platform engineers because they own services that other teams and customers depend on. When those services degrade, someone has to notice first and explain what happened.

The three observability pillars are easiest to understand by the question each one answers:

No single signal is enough. A metric can tell you error rates doubled, but it cannot tell you which request failed or why. A log line explains one failure but says nothing about how common it is. A trace shows that a downstream call added 800 ms, but only for the requests you sampled. Teams use all three because each one narrows the search for the next.

The main point of this guide is that you should collect each signal to answer a specific question. Collecting it just because a library emits it by default is not a good enough reason. The signals also need shared context, such as a common service name, request ID and trace ID, so you can move from an alert to the relevant traces and logs. The questions themselves should come from your SLOs, meaning what users actually need from the service. Telemetry that does not support a decision is cost without benefit.

What to measure first: RED, USE and latency distributions

Start with a small set of signals that you apply the same way everywhere. For every endpoint, track RED: request rate, error rate and latency distribution. For every shared resource that requests wait on, such as connection pools, queues, worker threads and CPU, track USE: utilization, saturation and errors. RED tells you that users are affected. USE usually tells you why. For example, a database pool at full utilization with a growing wait queue explains a latency spike that RED can only report.

Record latency as a histogram, not an average. Averages hide the slow tail. In an illustrative example, if 98 requests take 20 ms and 2 take 2 s, the mean is about 60 ms, which matches no real request. Percentiles such as p50, p95 and p99 show the tail. You also can't average percentiles: taking the mean of each instance's p99 doesn't give you the fleet-wide p99, because instances handle different volumes and see different distributions. Instead, sum the histogram bucket counts across instances first, then calculate the percentile from the combined counts. Bucket boundaries set the limit on precision. If your buckets are at 250 ms and 500 ms, any value between them is interpolated, so put boundaries close to your SLO threshold.

Watch cardinality. Every unique combination of label values becomes its own time series, which the backend has to store and query.

Copyfrom prometheus_client import Histogram, start_http_server

# One histogram covers RED: _count gives rate, the status_class label gives errors,
# and the buckets give the latency distribution.
# Buckets cluster around an assumed 300 ms SLO threshold so the percentile near it is precise.
REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency in seconds",
    labelnames=["route", "status_class", "region"],  # bounded labels only
    buckets=[0.05, 0.1, 0.2, 0.25, 0.3, 0.35, 0.5, 1.0, 2.5],  # +Inf is added automatically
)

REGION = "eu-west-1"  # placeholder, set once per deployment

def record(route_template: str, status_code: int, duration_s: float) -> None:
    # route_template is "/users/{id}", never the raw path "/users/81723".
    # User IDs and request IDs go in logs and trace attributes, not metric labels.
    status_class = f"{status_code // 100}xx"  # 2xx, 4xx, 5xx: a handful of values
    REQUEST_LATENCY.labels(
        route=route_template, status_class=status_class, region=REGION
    ).observe(duration_s)

if __name__ == "__main__":
    start_http_server(8000)  # exposes /metrics for scraping
    record("/users/{id}", 200, 0.021)
    record("/users/{id}", 503, 2.0)

# Fleet-wide p99 in PromQL: sum bucket rates across instances first, then compute.
#   histogram_quantile(0.99,
#     sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
#
# Wrong: this averages per-instance p99s, which is not a fleet p99.
#   avg by (route) (histogram_quantile(0.99,
#     sum by (le, route, instance) (rate(http_request_duration_seconds_bucket[5m]))))

A Prometheus histogram with bounded labels and buckets placed close to an assumed 300 ms SLO, plus PromQL that sums bucket counts across instances before computing p99 and the incorrect query that averages per-instance p99 values.

How it works: structured logs, distributed tracing and OpenTelemetry

Those unbounded values only become useful when every signal shares a common key. That key is the trace context.

Structured logs

Emit one JSON event per line with consistent field names: timestamp, severity, service.name, message, trace_id and span_id. Consistent names let a single query work across all services, and the IDs let you jump from a log line straight to the trace that produced it. A correlated line looks like {"severity":"ERROR","service.name":"checkout","message":"payment declined","trace_id":"4bf92f35...","span_id":"00f067aa..."}. Log at service boundaries and on failures. Logging inside hot loops multiplies volume and cost without adding diagnostic value.

Distributed tracing

A trace is a tree of spans. Each span is a timed operation that records its parent's ID. Context propagation keeps the tree connected across process boundaries. Over HTTP, the caller injects a header, commonly W3C traceparent, and the callee extracts it. Over a message queue, the same context travels in message metadata. If a proxy, gateway or consumer drops that context, nothing errors. The downstream work silently starts a new, orphaned trace, so verify propagation at every hop.

OpenTelemetry

OpenTelemetry is a vendor-neutral layer for producing and moving telemetry. Language SDKs and zero-code instrumentation generate the signals. The Collector receives, processes and exports them:

service → SDK → Collector → one or more backends

A minimal Collector pipeline declares three stages: an OTLP receiver, a batch processor and an exporter for each backend. Because exporters live in the Collector, you can switch backends by editing its config without changing application code.

Copy# Minimal OpenTelemetry Collector pipeline: receive OTLP, batch, fan out.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317   # SDKs send OTLP/gRPC here
      http:
        endpoint: 0.0.0.0:4318   # SDKs send OTLP/HTTP here

processors:
  batch: {}                      # group telemetry to cut export calls

exporters:
  # Primary backend. To switch vendors, change this block, not app code.
  otlp/primary:
    endpoint: telemetry-backend.example.com:4317
    headers:
      authorization: "Bearer ${env:BACKEND_API_TOKEN}"  # injected at runtime
  # Prints received telemetry to the Collector's own output for debugging.
  debug: {}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/primary, debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/primary]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/primary, debug]

A minimal Collector config that receives OTLP from application SDKs, batches it, and exports traces, metrics and logs to a backend, so changing backends only means editing the exporters block.

Defining SLIs, SLOs and error budgets

A service level indicator (SLI) is a ratio: good events divided by valid events. For a request-serving API, a good event might be a request that returns a non-5xx status and completes in under 300 ms. Valid events exclude traffic you shouldn't count, such as health checks or requests rejected for bad credentials. Defining the SLI as a ratio of events, rather than as an average, ties it to what users actually experienced.

A service level objective (SLO) sets a target for that ratio over a time window, for example 99.9% of valid requests are good over a rolling 30 days. A rolling window keeps the number current. A calendar month resets on the 1st, which can hide a bad final week. The error budget is what remains: 1 − SLO target, applied to the window.

Illustrative estimate (assumed numbers, not a benchmark): Budget = (1 − target) × window = (1 − 0.999) × 30 days × 1,440 min/day = 0.001 × 43,200 ≈ 43 minutes of full unavailability. Partial degradation consumes the budget more slowly. For example, failing 10% of requests uses it at one-tenth the rate.

Let the budget drive decisions. If plenty remains, keep shipping. If it is nearly spent, prioritize reliability work. Tune targets with care:

• Too tight: normal noise pages people, alert fatigue sets in, and releases freeze over failures users never noticed.
• Too loose: the SLO looks green while users experience real problems.
• Measured only at the server: the SLI misses failures in DNS, the load balancer or the client. Add client-side or synthetic probes where those layers matter.

Alerting that respects on-call

Page a human only when users are being hurt or soon will be. The SLO from the previous section already measures that, so alert on burn rate: how fast you are consuming the error budget relative to a steady pace. A burn rate of 1 uses exactly the whole budget over the SLO window. A burn rate of 10 uses it in a tenth of the time.

A single window forces a bad trade-off. A short window pages on brief blips, and a long window pages late and keeps firing after recovery. Multi-window alerts fix this by requiring both a long and a short window to exceed the threshold. The long window shows the problem is significant, and the short window shows it is still happening.

Example rule (illustrative): for a 99.9% SLO over 30 days, page when the burn rate exceeds 14.4 over the last hour and over the last 5 minutes. That corresponds to an error ratio above 1.44% and spends 2% of the monthly budget in one hour. Add a second, ticket-level rule at a burn rate of 6 over 6 hours and 30 minutes to catch slower degradation.

Keep a few targeted cause alerts where symptoms arrive too late, such as a disk that will fill within 24 hours or a certificate nearing expiry. Every alert, whether symptom or cause, must link to a runbook. An alert without a next step pushes the investigation onto whoever is woken up at 3 a.m.

Copy# Prometheus alerting rules. SLO: 99.9% of requests succeed over 30 days (budget 0.1%).
# Thresholds are illustrative; tune them to your own SLO and traffic.
groups:
  - name: api-slo-burn
    rules:
      # Page: burn rate > 14.4 over 1h (significant) AND over 5m (still happening).
      - alert: ApiErrorBudgetFastBurn
        expr: |
          (sum(rate(http_requests_total{job="api",code=~"5.."}[1h]))
            / sum(rate(http_requests_total{job="api"}[1h]))) > (14.4 * 0.001)
          and
          (sum(rate(http_requests_total{job="api",code=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m]))) > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "API is burning error budget 14.4x too fast"
          runbook_url: "RUNBOOK_BASE_URL/api-error-budget-fast-burn"
      # Ticket: slower degradation, burn rate > 6 over 6h AND 30m.
      - alert: ApiErrorBudgetSlowBurn
        expr: |
          (sum(rate(http_requests_total{job="api",code=~"5.."}[6h]))
            / sum(rate(http_requests_total{job="api"}[6h]))) > (6 * 0.001)
          and
          (sum(rate(http_requests_total{job="api",code=~"5.."}[30m]))
            / sum(rate(http_requests_total{job="api"}[30m]))) > (6 * 0.001)
        labels:
          severity: ticket
        annotations:
          runbook_url: "RUNBOOK_BASE_URL/api-error-budget-slow-burn"
      # Targeted cause alert: users feel a full disk only once it is too late.
      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 24 * 3600) < 0
        for: 30m
        labels:
          severity: ticket
        annotations:
          runbook_url: "RUNBOOK_BASE_URL/disk-filling"

Prometheus rules that page on a multi-window fast SLO burn and open tickets for slow burns and a disk projected to fill within 24 hours, each linked to a runbook via a placeholder URL.

Trade-offs, when to scale back, and alternatives

Every signal has a cost curve. High-cardinality metric labels such as user ID or request ID multiply the number of time series. Verbose logs grow linearly with traffic. Sampling 100% of traces mostly stores healthy, near-identical requests. All three raise storage and query costs faster than they add insight.

Sampling strategy is its own trade-off. Head-based sampling decides when a request starts, so it is cheap and stateless, but it cannot know whether that request will fail. Tail-based sampling waits for the complete trace and keeps errors and slow outliers. The cost is a stateful Collector tier that buffers spans in memory. If that tier is undersized, it drops data or becomes a failure point itself.

Auto-instrumentation handles HTTP and database calls well. It can miss async work, queue consumers and background jobs, which leaves gaps where context propagation is hardest. Logs can leak tokens or personal data, so enforce redaction and field allow-lists in the pipeline rather than relying on each developer. The monitoring pipeline often shares a network, cluster or region with the system it watches, so the two can fail together.

A minimal rollout order and its impact on cost and engineering time

Whichever of those alternatives you start from, add instrumentation in an order that pays off early and keeps costs predictable:

The cost comes from four places: writing the instrumentation, storing the data, querying it, and the friction of keeping secrets and personal data out of logs and span attributes. Label cardinality is usually the cost that grows fastest.

Illustrative estimate (assumed numbers, not a benchmark): series = routes × status classes × instances × histogram buckets. With 40 routes, 5 status classes, 10 instances and 12 buckets, one latency metric produces 24,000 series. If you add a user_id label with 50,000 users, the same metric produces 50,000 times as many series.

The time savings come from correlation. An engineer can follow an SLO burn alert to a slow trace, then use its trace_id to find the exact log lines, with no guessing at each step. Paging on SLO burn rather than on raw thresholds means pages are fewer and usually actionable. Instrumenting with OpenTelemetry APIs and routing data through its Collector means changing backends is mostly a matter of updating exporter configuration, without touching application code.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to backend monitoring, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.