What is a burn rate alert, and why do static error-rate thresholds fail?

A service level indicator (SLI) is a ratio of good events to total events, such as successful requests divided by all requests. A service level objective (SLO) is the target for that ratio over a window, for example 99.9% over 30 days. The error budget is what's left: 0.1% of requests may fail. The burn rate is how fast you spend that budget:

burn rate = observed error ratio / (1 − SLO)

Derived examples (99.9% SLO, 30-day window)

  • An error ratio of 0.1% is a burn rate of 1. The budget runs out exactly at the end of the window.
  • A burn rate of 14.4 means a 1.44% error ratio. Sustained for 1 hour, it consumes 14.4 × 1h / 720h = 2% of the monthly budget.
  • A 0.5% error ratio is a burn rate of 5. It exhausts the budget in 30 / 5 = 6 days.

Why static thresholds fail

Take an illustrative rule of "page if errors exceed 5% for 5 minutes." The 0.5% leak above never triggers it, so the budget is gone within a week and nobody gets paged. A 2-minute spike to 10% errors is a burn rate of 100. It may trigger the rule, yet it spends only about 0.46% of the budget, which is not worth waking someone up for. A fixed threshold measures how bad things are right now. It says nothing about whether you will miss the objective.

This approach fits backend and platform teams that run request-serving APIs with an agreed SLO. It also helps product owners and incident leads who decide what to do when the budget runs low, such as freezing releases or reprioritizing reliability work.

How do you define availability and latency SLIs for a backend API?

An SLI is a ratio: good events divided by valid events. Before writing a single rule, decide which requests count as valid and which count as good. These choices set the error budget, so record them in the SLO document. Do not leave them implicit in a PromQL expression.

Request class Counts as valid? Counts as good? Reasoning
2xx / 3xx Yes Yes The service did its job
5xx Yes No Server-side failure
4xx (except 429) Usually yes Yes The caller's mistake should not spend your budget
429 Yes Depends If you throttle legitimate traffic, users see an outage
Server-observed timeouts Yes No Late is the same as failed for the caller
Health checks, probes No — High-volume, trivial requests inflate the success ratio

Latency as a ratio, not a quantile

Define latency as the fraction of requests faster than a threshold. Compute it from histogram counters: the rate of _bucket{le="0.3"} divided by the rate of _count. Precomputed quantiles cannot be averaged across instances in any meaningful way. Counters sum correctly across instances. The threshold must match an existing le boundary. A 250 ms objective needs a 0.25 bucket. Otherwise Prometheus can only interpolate, and the SLI becomes an approximation.

What server-side metrics miss

Your application only counts requests that reach it. These failures never get recorded:

  • DNS failures
  • TLS errors
  • Load balancer rejections
  • Network drops

If those matter, measure at the edge with load balancer metrics, or from the client side with synthetic probes or real-user telemetry.

# Recording rules for availability and latency SLIs (good / valid).
# Metric and label names are placeholders; match your instrumentation.
groups:
  - name: api-sli
    rules:
      # Availability: valid = all requests except health checks and probes.
      # Good = everything except 5xx and 429. Other 4xx count as good.
      # This assumes server-side timeouts are recorded as 5xx (e.g. 504).
      - record: job:sli_availability:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="api", handler!~"/healthz|/readyz", code!~"5..|429"}[5m]))
          /
          sum by (job) (rate(http_requests_total{job="api", handler!~"/healthz|/readyz"}[5m]))

      # Latency: fraction of valid requests completing in 300 ms or less.
      # le="0.3" must be a real bucket boundary in the histogram.
      # Summing counters across instances is correct; averaging quantiles is not.
      - record: job:sli_latency_300ms:ratio_rate5m
        expr: |
          sum by (job) (rate(http_request_duration_seconds_bucket{job="api", handler!~"/healthz|/readyz", le="0.3"}[5m]))
          /
          sum by (job) (rate(http_request_duration_seconds_count{job="api", handler!~"/healthz|/readyz"}[5m]))

Prometheus recording rules that encode the valid/good decisions from the table: health checks are excluded, 5xx and 429 are bad, and latency is computed as a bucket-to-count ratio at an existing le boundary.

How do multi-window, multi-burn-rate alerts work?

A burn rate is how fast you spend error budget relative to the SLO period. At 1x, you spend the whole budget exactly at the end of the period. At 14.4x on a 30-day SLO, you spend it in about two days.

A single long window, such as 6 hours, smooths out noise. It also reacts slowly to a sudden outage, and it keeps firing for hours after the fix because the bad minutes stay inside the window. A single short window, such as 5 minutes, reacts fast but pages on every brief spike.

The fix is to require both. The long window shows that the burn has lasted long enough to matter. The short window, about 1/12 the long window's length, shows that the burn is still happening. When the incident ends, the short window drops below threshold within minutes, and the alert clears even while the long window is still elevated.

A common starting policy for a 30-day SLO:

Severity Burn rate Long / short window Budget consumed in long window
Page 14.4x 1h / 5m 2%
Page 6x 6h / 30m 5%
Ticket 3x 1d / 2h 10%
Ticket 1x 3d / 6h 10%

The budget column is the input you choose. The burn rate is derived from it:

burn rate = budget fraction consumed × SLO period ÷ long window

For a 30-day (720h) period, 0.02 × 720 ÷ 1 = 14.4.

For a 28-day (672h) period, the same budget percentages give the following rates (illustrative estimate):

  • 13.44x for the 1h window
  • 5.6x for the 6h window
  • 2.8x for the 1d window
  • about 0.93x for the 3d window
              +------------------------------------+
              | SLI: error ratio = bad / total     |
              +------------------------------------+
                  |                               |
                  v                               v
   +----------------------------+  +----------------------------+
   | long window (1h)           |  | short window (5m = 1h/12)  |
   | burn rate > 14.4x ?        |  | burn rate > 14.4x ?        |
   | "burned long enough"       |  | "still burning now"        |
   +----------------------------+  +----------------------------+
                  |                               |
                  +-------------------------------+
                                  |
                                  v
                           +-------------+
                           |     AND     |---> fire page
                           +-------------+
  after fix: short window drops in minutes ---> AND false ---> clears

  tier    burn   long/short  budget   burn = budget x period / long
  ------  -----  ----------  ------   ------------------------------
  page    14.4x  1h / 5m     2%       0.02 x 720h / 1h  = 14.4
  page    6x     6h / 30m    5%       0.05 x 720h / 6h  = 6
  ticket  3x     1d / 2h     10%      0.10 x 720h / 24h = 3
  ticket  1x     3d / 6h     10%      0.10 x 720h / 72h = 1
  28-day SLO: swap 720h for 672h ---> 13.44x, 5.6x, 2.8x, ~0.93x

An alert fires only when both the long window (sustained burn) and the 1/12-length short window (ongoing burn) exceed the threshold; each tier's burn rate is derived from budget fraction x SLO period / long window.

How do you implement burn rate alerts in Prometheus with recording rules?

Step 1: record the error ratio at every window. Write one recording rule per window: 5m, 30m, 1h, 2h, 6h, 1d and 3d. Each one evaluates sum by (service, route) (rate(http_requests_total{code=~"5.."}[w])) / sum by (service, route) (rate(http_requests_total[w])). Name them consistently, for example service_route:request_errors:ratio_rate1h, so the alert expressions read clearly. Keep the service and route labels on every rule so the left and right sides of each comparison match. Compute each window from raw counters. Averaging short-window ratios gives the wrong answer when traffic is uneven.

Step 2: write one alert per tier.

  • Page: (ratio_rate1h > 14.4 * 0.001 and ratio_rate5m > 14.4 * 0.001) or (ratio_rate6h > 6 * 0.001 and ratio_rate30m > 6 * 0.001), with severity="page".
  • Ticket: pair 1d with 2h, and 3d with 6h, with severity="ticket".

In both, 0.001 is the error budget for a 99.9% SLO.

Step 3: keep for short or omit it. The long window already smooths out noise, and the short window clears the alert quickly once the problem stops. A for of more than a minute or two only delays detection. A short for is still useful to absorb a missed scrape.

Step 4: write annotations that give context. Include:

  • the current burn rate ({{ $value }})
  • the window pair that fired
  • how much budget that pace consumes, for example "2% of the 28-day budget per hour"

Latency variant. Define the "bad" ratio as 1 - (rate(..._bucket{le="0.3"}[w]) / rate(..._count[w])). This only works if a histogram bucket boundary sits exactly at your latency threshold.

Budget remaining. For dashboards, use 1 - (ratio_rate28d / 0.001). This value shows how much budget is left for the period. It does not tell you whether to act right now, which is the alert's job. Keep it off the pager.

groups:
  - name: slo-http-errors-recording
    rules:
      # One rule per window. Repeat this pattern for 30m, 2h, 6h, 1d, 3d (and 28d for dashboards).
      # Always compute from raw counters. Never average shorter-window ratios.
      - record: service_route:request_errors:ratio_rate5m
        expr: |
          sum by (service, route) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (service, route) (rate(http_requests_total[5m]))
      - record: service_route:request_errors:ratio_rate1h
        expr: |
          sum by (service, route) (rate(http_requests_total{code=~"5.."}[1h]))
          /
          sum by (service, route) (rate(http_requests_total[1h]))
  - name: slo-http-errors-alerts
    rules:
      - alert: ErrorBudgetBurnPage
        # Dividing by the budget (0.001 for a 99.9% SLO) makes $value the burn rate.
        # Both sides keep service/route labels, so 'and' matches series one-to-one.
        expr: |
          (
            service_route:request_errors:ratio_rate1h / 0.001 > 14.4
            and
            service_route:request_errors:ratio_rate5m / 0.001 > 14.4
          )
          or
          (
            service_route:request_errors:ratio_rate6h / 0.001 > 6
            and
            service_route:request_errors:ratio_rate30m / 0.001 > 6
          )
        # The long window already filters noise. 1m only absorbs a missed scrape.
        for: 1m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.service }} {{ $labels.route }} burning error budget at {{ $value | printf "%.1f" }}x'
          description: 'Window pair 1h/5m (14.4x, about 2% of the 28-day budget per hour) or 6h/30m (6x, about 0.9% per hour). SLO 99.9%, budget 0.1%.'

Prometheus rule file that records error ratios per window from raw counters and pages when the 1h/5m or 6h/30m burn-rate pair exceeds its threshold. The ticket tier repeats the alert pattern with the 1d/2h and 3d/6h pairs.

How do you route and test SLO alerts before relying on them?

Alertmanager should not need to understand burn rates. It only needs the severity label your rules attach. Add one child route under the root that matches severity="page" and sends to the on-call receiver. Add a second that matches severity="ticket" and sends to a queue-backed receiver, such as a webhook that opens tickets. Give the ticket route a long repeat_interval, because re-notifying a queue every few hours only creates duplicates.

A fast burn usually crosses the slow-burn thresholds as well. Without inhibition, the responder gets paged and the team also receives a ticket for the same incident. Add an inhibit rule with source_matchers on severity="page", target_matchers on severity="ticket", and equal: [service, slo]. The equal list matters. If you leave out slo, an availability page would silence an unrelated latency ticket for the same service.

Test the rules with promtool test rules before trusting them. Write synthetic input_series for the request and error counters using the expanding notation, such as 0+100x720 for steady traffic. Then cover three cases:

  • Fast spike: errors jump to a high rate for a short period. Assert that the page alert fires once its for duration has elapsed.
  • Slow leak: errors rise slightly above budget for hours. Assert that the ticket fires and the page never does.
  • Recovery: errors return to baseline. Assert that exp_alerts is empty shortly afterward. This shows the short window clears the alert, even though the long window still reports elevated burn.

Run these tests in CI so that a threshold change fails the build before it reaches on-call.

# alertmanager.yml: route on the severity label only. Alertmanager knows nothing about burn rates.
route:
  receiver: default-null          # fallback for anything unmatched
  group_by: [alertname, service, slo]
  group_wait: 30s
  group_interval: 5m
  routes:
    # Fast-burn tier: wake someone up.
    - matchers:
        - severity="page"
      receiver: oncall
      repeat_interval: 1h
    # Slow-burn tier: open a ticket once, don't spam the queue.
    - matchers:
        - severity="ticket"
      receiver: ticket-queue
      repeat_interval: 72h

inhibit_rules:
  # A firing page suppresses the overlapping ticket for the same SLO.
  # Include slo in equal so an availability page cannot hide a latency ticket.
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: [service, slo]

receivers:
  - name: default-null
  - name: oncall
    webhook_configs:
      - url:    # placeholder
        send_resolved: true
  - name: ticket-queue
    webhook_configs:
      - url:    # placeholder
        send_resolved: false

An Alertmanager config that sends severity="page" alerts to on-call and severity="ticket" alerts to a queue webhook with a long repeat interval, and uses an inhibit rule keyed on service and slo so a page suppresses the matching ticket.

When should you avoid burn rate alerts, and what are the alternatives?

Burn rate math assumes enough requests for the error ratio to mean something. On a low-traffic API, it doesn't. As an illustrative estimate, take a 99.9% SLO and two requests in a five-minute window. One failure produces a 50% error ratio and a burn rate of 500, so a single bad request pages someone.

Each common mitigation trades away some visibility:

  • Minimum request count: Suppressing the alert below a request floor also hides real outages during quiet hours.
  • Longer short windows: These smooth out noise, but detection and recovery both slow down.
  • Synthetic traffic: It inflates the denominator, which dilutes real user errors. It also only exercises the paths the probe calls.
  • Aggregating endpoints: A fully broken low-volume route disappears inside a healthy high-volume one.

The same tension applies to aggregation level. A service-wide SLO needs few rules and has enough traffic, but it masks route-level failures. Per-route SLOs are precise, but they multiply rules and bring back the low-traffic problem. A workable middle ground is a few route groups by criticality, such as writes, reads, and admin.

Approach Detects Weakness
Static error-rate threshold Sudden error spikes Ignores the error budget, so it either flaps or misses slow leaks
Single-window burn rate Budget consumption Either slow to fire or slow to reset
Saturation and dependency alerts Likely causes (queue depth, pool exhaustion) Can fire with no user impact
Synthetic probes Total outages, even with zero traffic Narrow coverage of real request paths

Burn rate alerts tell you the budget is draining, not why. Pair them with cause-oriented signals so the person paged has a place to start.

What does SLO alerting cost to build and maintain, and where does it save engineering time?

The main build cost is rule sprawl. Each service multiplies SLIs by windows by alert tiers, so hand-written YAML drifts quickly. Most teams end up generating rules from a per-service SLO spec using templates or a config generator. That adds tooling you have to own: a generator, a schema, and CI checks that run promtool check rules and rule unit tests on the generated output.

Illustrative estimate (assumed numbers, not a benchmark). Use this formula: rules = services × SLIs × (windows + alert tiers).

  • Assume 20 services, 2 SLIs each (availability and latency), 6 recorded windows, and 3 alert tiers.
  • That gives 20 × 2 × (6 + 3) = 360 rules.

Nobody should review 360 rules by hand. You review the generator and the specs instead.

Long ranges cost the most to query and are the most fragile:

  • Retention. 3d and 30-day windows need retention that covers them.
  • Scrape gaps and restarts. Gaps and restarts drop increments that rate() cannot recover. rate() handles counter resets, but sparse samples across a restart still skew the ratio.
  • Query load. Recording short windows and building longer ones from them keeps evaluation cheap.

Don't copy the 30-day burn-rate factors (14.4, 6, 1) unchanged to a 7-day or 28-day SLO. Each factor encodes a budget fraction over a specific period, so recompute them for your own period.

The savings come from fewer false pages and from catching slow leaks that never cross a static threshold. Readable expressions and clear severity routing also shorten triage.

All of this depends on targets that product and service owners agreed to. Without that agreement, slow-burn tickets become noise that nobody acts on.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to burn rate alerts prometheus, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.