What are SLO burn rate alerts and how do they differ from error-rate thresholds?

An SLO burn rate alert fires on how fast you are spending your error budget, not on whether the error rate crosses a fixed line. That requires a request-based SLI: the fraction of good events over valid events in a window.

For an API, decide what "valid" and "good" mean before writing any rules:

  • 5xx: Count as bad. The server failed.
  • 4xx: Usually excluded or counted as good, because the client made the mistake. A bad deploy that returns 400 to valid requests will then not register.
  • 429: This is a policy call. If rate limiting protects you from abuse, exclude it. If it reflects capacity you owe paying clients, count it as bad.
  • Timeouts and dropped connections: Count as bad. They may never produce a status code at the server, so measure them at the load balancer or client where possible.

Both choices have a cost. An excluded error class can hide a real outage, and an included one can page on client misbehaviour.

For latency, the SLI is the share of requests faster than a fixed threshold. You read this from the histogram bucket whose le matches the threshold (for example le="0.3") divided by the _count series. The threshold must be an existing bucket boundary, or the result is interpolated rather than exact.

Approach Brief spike at low traffic Slow 0.5% leak for days
Static threshold (error rate > 5% for 5m) Pages Never fires
Burn-rate alert Ignored if budget impact is small Fires before budget runs out
# Prometheus recording rules for request-based SLIs on one API.
# Policy encoded here: 5xx and 429 are bad; other 4xx are not valid events.
# Timeouts that never reach the server are not visible in these metrics.
groups:
  - name: checkout-api-sli
    rules:
      # Valid events: everything except client errors, but keep 429
      # because this policy treats throttling as owed capacity.
      - record: job:slo_valid_requests:rate5m
        expr: |
          sum by (job) (
            rate(http_requests_total{job="checkout-api", code=~"[1235]..|429"}[5m])
          )

      # Bad events: server failures plus throttled requests.
      - record: job:slo_bad_requests:rate5m
        expr: |
          sum by (job) (
            rate(http_requests_total{job="checkout-api", code=~"5..|429"}[5m])
          )

      # Availability error ratio (1 - SLI). Burn-rate alerts compare this to the budget.
      - record: job:slo_errors_per_request:ratio_rate5m
        expr: job:slo_bad_requests:rate5m / job:slo_valid_requests:rate5m

      # Latency SLI: share of requests completing within 300ms.
      # le="0.3" must be a real bucket boundary in the histogram, otherwise
      # this rule returns no data (it does not interpolate).
      - record: job:slo_latency_good_ratio:rate5m
        expr: |
          sum by (job) (
            rate(http_request_duration_seconds_bucket{job="checkout-api", le="0.3"}[5m])
          )
          /
          sum by (job) (
            rate(http_request_duration_seconds_count{job="checkout-api"}[5m])
          )

Prometheus recording rules that encode the error-classification policy as label matchers and derive availability and latency SLIs, the inputs a burn-rate alert compares against the error budget.

How does error budget burn rate work?

The error budget is the fraction of requests you are allowed to fail: error budget = 1 − SLO target. A 99.9% target leaves a budget of 0.1%. Burn rate measures how fast you are spending that budget relative to the sustainable pace:

burn rate = observed error ratio / (1 − SLO target)

At a burn rate of 1, you consume exactly the full budget by the end of the SLO window. At a burn rate of 10, you consume it in one tenth of the window. Burn rate is useful because a single number tells you both how bad things are and how long you have left: time to exhaustion = SLO window / burn rate.

Deriving the alert factor

Choose how much of the budget you will tolerate losing before someone is notified. The factor follows from that choice:

factor = (budget fraction × SLO window) / alert window

Illustrative estimate (assumed numbers, not a benchmark). Assume a 99.9% target over 30 days (720 hours):

Budget lost before alert Alert window Factor Error ratio threshold Time to exhaustion
2% 1 hour 0.02 × 720 / 1 = 14.4 14.4 × 0.001 = 1.44% 720 / 14.4 = 50 hours
5% 6 hours 0.05 × 720 / 6 = 6 6 × 0.001 = 0.6% 720 / 6 = 120 hours (5 days)

These factors are not universal constants. The factor depends on the SLO window. With a 7-day window (168 hours), "2% in 1 hour" gives a factor of 3.36, not 14.4. The error ratio threshold depends on the target. At 99.99%, a factor of 14.4 corresponds to 0.144% errors. At low traffic, a handful of failed requests can produce that ratio. Recompute both values whenever you change either the target or the window.

How do you implement multiwindow, multi-burn-rate alerts with Prometheus recording rules?

Precompute one error ratio per window so every alert expression reads from the same source. Name the rules after the window, such as service:api_errors_per_request:ratio_rate5m through ratio_rate3d. Aggregate each rule by the same label set, typically service alone. If two windows carry different labels, the and that joins them silently matches nothing and the alert never fires.

Each alert needs both of its windows above the threshold. The long window shows that enough budget has burned to matter. The short window, about 1/12 of the long one, confirms the burn is still happening, so the alert resolves within minutes of recovery instead of staying active until the long window drains.

Burn factor Long window Short window Budget consumed (30-day SLO) Action
14.4 1h 5m 2% Page
6 6h 30m 5% Page
3 1d 2h 10% Ticket
1 3d 6h 10% Ticket

The threshold in each expression is the burn factor multiplied by the error budget, for example 14.4 * 0.001 for a 99.9% target.

To stop a few failures on a quiet service from paging someone, add a traffic floor. Append and with a request-rate condition over the long window, such as requiring more than one request per second on average over the past hour. Pick the floor from the service's real off-peak traffic.

The cost is in series count. Seven windows for each SLO is seven recording rules, and each rule produces one series for every combination of labels it keeps. Adding a per-endpoint or per-customer label multiplies those series by the cardinality of that label. Keep fine-grained labels off the SLO rules and use them in dashboards for diagnosis.

groups:
  - name: slo-api-recording
    rules:
      # One ratio per window, all aggregated "by (service)" so alert joins on identical label sets.
      - record: service:api_errors_per_request:ratio_rate5m
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))
      - record: service:api_errors_per_request:ratio_rate30m
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[30m])) / sum by (service) (rate(http_requests_total[30m]))
      - record: service:api_errors_per_request:ratio_rate1h
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[1h])) / sum by (service) (rate(http_requests_total[1h]))
      - record: service:api_errors_per_request:ratio_rate2h
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[2h])) / sum by (service) (rate(http_requests_total[2h]))
      - record: service:api_errors_per_request:ratio_rate6h
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[6h])) / sum by (service) (rate(http_requests_total[6h]))
      - record: service:api_errors_per_request:ratio_rate1d
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[1d])) / sum by (service) (rate(http_requests_total[1d]))
      - record: service:api_errors_per_request:ratio_rate3d
        expr: sum by (service) (rate(http_requests_total{code=~"5.."}[3d])) / sum by (service) (rate(http_requests_total[3d]))
  - name: slo-api-alerts
    rules:
      - alert: APIErrorBudgetFastBurn
        # 99.9% target -> budget 0.001. Long AND short window must both exceed burn factor * budget; floor of 1 req/s over 1h.
        expr: |
          (
            (service:api_errors_per_request:ratio_rate1h > (14.4 * 0.001) and service:api_errors_per_request:ratio_rate5m > (14.4 * 0.001))
            or (service:api_errors_per_request:ratio_rate6h > (6 * 0.001) and service:api_errors_per_request:ratio_rate30m > (6 * 0.001))
          )
          and on (service) sum by (service) (rate(http_requests_total[1h])) > 1
        labels:
          severity: page
      - alert: APIErrorBudgetSlowBurn
        expr: |
          (
            (service:api_errors_per_request:ratio_rate1d > (3 * 0.001) and service:api_errors_per_request:ratio_rate2h > (3 * 0.001))
            or (service:api_errors_per_request:ratio_rate3d > (1 * 0.001) and service:api_errors_per_request:ratio_rate6h > (1 * 0.001))
          )
          and on (service) sum by (service) (rate(http_requests_total[1d])) > 1
        labels:
          severity: ticket

Prometheus rule file that records per-service error ratios for seven windows with identical labels, then pages on fast burns and opens tickets on slow burns only when both the long and short windows exceed the threshold and traffic clears a minimum request-rate floor.

Routing burn-rate tiers through Prometheus Alertmanager

Alerting rules emit the tier as a severity label: page for the fast-burn pair (for example 1h/5m windows) and ticket for the slow-burn pair (6h/30m or 3d/6h). Alertmanager routes on that label, so the paging decision lives in rule definitions that are reviewed like code.

scrape -> recording rules -> alerting rules -> Alertmanager
          (ratio per window)  (severity=page|    | group_by: service, slo
                               severity=ticket)  | inhibit: page mutes ticket
                                                 +--> pager receiver
                                                 +--> ticket receiver

Routes and grouping

Set group_by to service and slo so that a fast-burn and a slow-burn firing for the same objective arrive as one coherent notification, not a stream of per-instance noise. Add child routes that match severity="page" to the pager receiver and severity="ticket" to the ticket queue.

Inhibition

Add an inhibit rule with a source matching severity="page" and a target matching severity="ticket". Set equal on service and slo. While the page is active, the duplicate ticket is suppressed. A page for checkout latency never mutes a ticket for search availability.

for duration

Keep for at zero or a minute or two. The short window already confirms the burn is ongoing, and the long window already filters out spikes. A long for only delays a page that has already been validated twice.

Annotations

Template these fields so that whoever is on call can act without opening a query editor:

  • The SLO target and window.
  • The current burn rate from $value.
  • The budget it consumes, for example "at this rate, the 30-day budget is gone in about 2 days."
  • A runbook pointer.
  • A dashboard link filtered to the alert's service.
# alertmanager.yml: route SLO burn-rate alerts by the severity label set in alerting rules
route:
  receiver: ticket-queue           # fallback: unrouted alerts become tickets, never silent
  group_by: ['service', 'slo']     # one notification per objective, not per instance
  group_wait: 30s                  # brief pause so fast- and slow-burn alerts land together
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Fast-burn pair (e.g. 1h/5m windows): wake someone up.
    - matchers: ['severity="page"']
      receiver: pager
      repeat_interval: 1h
    # Slow-burn pairs (e.g. 6h/30m, 3d/6h): work it during business hours.
    - matchers: ['severity="ticket"']
      receiver: ticket-queue
      repeat_interval: 24h

inhibit_rules:
  # An active page mutes the matching ticket, but only for the same service AND SLO.
  # A checkout-latency page never suppresses a search-availability ticket.
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: ['service', 'slo']

receivers:
  - name: pager
    webhook_configs:
      - url: ''   # placeholder endpoint
        send_resolved: true
  - name: ticket-queue
    webhook_configs:
      - url: ''  # placeholder endpoint
        send_resolved: true

An Alertmanager configuration that groups burn-rate alerts by service and SLO, sends severity=page to the pager and severity=ticket to the ticket queue, and inhibits a ticket while a page for the same service and SLO is firing.

How do you test SLO alert rules before trusting them?

Burn-rate expressions are easy to get subtly wrong: a window mismatch between recording rules, a ratio inverted, or a for clause that delays the page past the point where it helps. You should not wait for production traffic to find these mistakes. promtool test rules evaluates your recording and alerting rules against synthetic series and checks which alerts are firing at specific evaluation times.

Unit-test each tier with synthetic series

Write a test file that loads the same rule files you deploy. Declare input series for the request and error counters using expanding notation, so each series is a counter that grows at a fixed rate. Then build scenarios that target each tier:

  • Fast burn: a sharp error spike. Assert that the fast-tier alert fires within the expected number of minutes and that the slow tier does not fire yet.
  • Slow burn: a modest, sustained error rate over many hours. Assert that only the slow tier fires.
  • Recovery: return errors to zero. Assert that each alert resolves once the short window clears, not hours later.

Pick a sample interval and series length that cover your longest window. A test that only spans one hour cannot show anything about a 6-hour or 3-day window.

Replay a known incident

Export the raw counter values for a past incident from Prometheus and paste them into a test as input series. Check when the rules would have paged. If that is too late to matter, or the rules page during a blip nobody acted on, change the thresholds now and keep the test as a regression case.

Each threshold change then comes with evidence attached. On-call engineers spend less time arguing about whether a page was real, and fewer pages that later prove to be false need investigating.

# slo_test.yml -- run with: promtool test rules slo_test.yml
# Assumes slo_rules.yml defines APIErrorBudgetFastBurn (1h + 5m windows, 14.4x burn
# for a 99.9% SLO) with labels {severity: page, slo: api-availability}, no annotations.
rule_files:
  - slo_rules.yml          # the same file you deploy
evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      # Steady 10 req/s of successful traffic for 3 hours.
      - series: 'http_requests_total{job="api",code="200"}'
        values: '0+600x180'
      # No errors for 60m, then about 9% errors for 30m, then errors stop (recovery).
      - series: 'http_requests_total{job="api",code="500"}'
        values: '0x59 0+60x30 1800x90'

    alert_rule_test:
      # Before the spike: the fast tier must be silent.
      - eval_time: 55m
        alertname: APIErrorBudgetFastBurn
        exp_alerts: []
      # 15 minutes into the spike: the fast tier must be paging.
      - eval_time: 75m
        alertname: APIErrorBudgetFastBurn
        exp_alerts:
          - exp_labels:
              severity: page
              slo: api-availability
      # 10 minutes after errors stop the 5m window is clean, so the alert must resolve.
      - eval_time: 100m
        alertname: APIErrorBudgetFastBurn
        exp_alerts: []

This promtool rule test feeds synthetic request and error counters into the deployed rules and asserts that the fast-burn alert is silent before an error spike, fires during it and resolves soon after recovery. A slow-burn case needs its own test whose series run longer than the 6h window.

When should you avoid burn-rate alerting or adapt it?

Burn-rate alerting works best for teams with steady request volume and an SLO that product and engineering have actually agreed on. Outside those conditions, adapt it or drop it.

  • Low-traffic endpoints. At 50 requests per hour, two failures are a 4% error rate, which looks like a fast burn against a 99.9% target and pages someone. A minimum-request guard prevents that, but it also hides the 3 a.m. outage where traffic drops below the guard because the endpoint is broken. You can aggregate low-volume endpoints into one SLO per service or user journey. Another option is to run synthetic requests at a fixed rate so the denominator never collapses.
  • Long windows and data integrity. A 30-day budget needs 30 days of retention. It is also exposed to scrape gaps and restarts. If a pod dies before its last increments are scraped, those errors disappear. During a gap, rate() has too few samples and returns nothing, and an empty result never fires. Add absent() or staleness alerts on the SLI series themselves so missing data does not look healthy.
  • Rolling budgets. A slow-burn ticket can stay open for days after the fix ships, until the bad hours roll out of the window. Close the ticket on the fix, not on the alert resolving.
  • Server-side blind spots. If the load balancer, DNS or a client-side timeout drops a request, your server never records it. Pair server SLIs with black-box probes or load-balancer metrics.

What are the alternatives to SLO burn rate alerts, and what do they cost?

Each option answers a different question, so the useful comparison covers what each one misses and what it costs to run.

Approach Signal quality Noise Setup effort Evaluation cost
Static error-rate threshold Ignores the SLO and traffic volume High at low traffic, flaps near the line One expression Negligible
Single-window burn rate Tied to the error budget, but slow to fire or slow to reset Moderate Low Low
Multiwindow, multi-burn-rate Measures user impact with fast reset Low Recording rules plus per-SLO factors One series per window per SLI label set
Cause-based (disk filling, certificate expiry, queue saturation) Predicts failures before users notice; says nothing about impact Low if predictive Low per rule, many rules Low
Synthetic or client-side probes Sees failures the server never logs Sparse samples, noisy at low frequency Probe infrastructure External, plus a few series

Burn-rate alerts complement cause-based alerts rather than replace them. A certificate expiring in ten days burns no budget until it expires, and then it burns all of it at once. Keep cause-based rules, and route most of them to tickets instead of pages.

Illustrative estimate (assumed numbers, not a benchmark)

  • Added series = SLIs × windows × label combinations. With 20 SLIs, 6 windows and 1 label set, that is 120 series.
  • Payback in weeks = authoring hours ÷ (noisy pages avoided per week × hours per page). With 16 hours of authoring, 10 pages avoided per week and 0.33 hours per page, that is 16 ÷ 3.3 ≈ 4.8 weeks.

Add ongoing maintenance to the estimate. You must recompute the burn-rate factors whenever an SLO target or window changes, and you must review rules whenever a label is added or removed.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to slo burn rate alerts, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.