What is an SLO burn rate alert, and how do you define the SLI for an API?

An SLO burn rate alert fires when you are consuming error budget faster than your SLO allows, rather than when an error rate crosses an arbitrary line. Before you can measure burn, you need an SLI that counts events correctly.

For availability, define the SLI as good requests divided by valid requests. Valid requests are the ones your service is responsible for. Exclude client-caused 4xx responses such as malformed input or failed authentication, because a buggy client should not drain your budget. Good requests are the valid ones that did not return a 5xx. Decide case by case whether 429s belong in the numerator or the denominator.

For latency, define the SLI as the fraction of requests faster than a threshold. Take this from histogram bucket counters. The _bucket series with le set to your threshold, divided by _count, gives the share of fast requests. The threshold must match an existing bucket boundary, so choose your buckets with the SLO in mind.

Both SLIs are ratios of counters, and that is the reason they work. You can sum counters across instances and take rate() over any window, and the resulting ratio stays mathematically valid. Averaging per-instance p99 values does not work, because quantiles do not compose.

These SLIs are measured on the server side. They do not see failures at the load balancer, in DNS, or from client timeouts that never reach your process. Your alert can only be as good as the SLI behind it.

This approach suits backend and platform teams that have real request volume and an SLO they have already agreed on.

# Recording rules for ratio-of-counters SLIs.
# Assumes http_requests_total{job, code} and the histogram
# http_request_duration_seconds with a bucket at le="0.3".
groups:
  - name: api-sli
    rules:
      # Valid requests exclude client-caused 4xx.
      # 429 is excluded here too. Move it into the error set
      # if your own rate limiting counts against the SLO.
      - record: job:sli_valid_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{code!~"4.."}[5m]))

      # Good requests are the valid requests that did not return a 5xx.
      - record: job:sli_good_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{code!~"4..|5.."}[5m]))

      # Availability SLI. Sum rates across instances first, then divide.
      # Never average per-instance ratios.
      - record: job:sli_availability:ratio_rate5m
        expr: |
          job:sli_good_requests:rate5m
          /
          job:sli_valid_requests:rate5m

      # Latency SLI: share of requests completing in 300ms or less.
      # The le value must exactly match an existing bucket boundary.
      - record: job:sli_latency_fast:ratio_rate5m
        expr: |
          sum by (job) (rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
          /
          sum by (job) (rate(http_request_duration_seconds_count[5m]))

      # The same ratio over a longer window. Counters compose, so only the
      # range changes. Burn rate alerts reuse this pattern at several windows.
      - record: job:sli_availability:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{code!~"4..|5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total{code!~"4.."}[1h]))

Prometheus recording rules that define the availability SLI as good requests over valid requests and the latency SLI as the share of histogram observations under a 300ms bucket, both built from summed counter rates so they aggregate correctly across instances and windows.

How do error budget and burn rate work? Deriving the 14.4, 6, 3 and 1 thresholds

The error budget is the share of requests your SLO allows to fail: budget = 1 − target. A 99.9% target leaves a budget of 0.001, or 0.1% of requests. Burn rate = observed error ratio ÷ budget. A burn rate of 1 means you fail exactly 0.1% of requests, so you use the whole budget by the end of the SLO window. A burn rate of 10 uses it ten times faster.

To decide when to alert, choose how much budget you can afford to lose in a given window:

share of budget consumed = burn rate × alert window ÷ SLO window

Rearranged: burn rate = share × SLO window ÷ alert window

Illustrative estimate (assumed numbers, not a benchmark): 99.9% target, 30-day window (720 h).

  • 2% of budget in 1 h: 0.02 × 720 ÷ 1 = 14.4
  • 5% in 6 h: 0.05 × 720 ÷ 6 = 6
  • 10% in 1 d: 0.10 × 720 ÷ 24 = 3
  • 10% in 3 d: 0.10 × 720 ÷ 72 = 1

The burn rate threshold depends only on the windows. The target decides which error ratio triggers it:

Target / SLO window Budget 1 h threshold (2% of budget) Error ratio that fires
99.9% / 30 d 0.1% 14.4 1.44%
99% / 30 d 1% 14.4 14.4%
99.99% / 30 d 0.01% 14.4 0.144%
99.9% / 7 d 0.1% 3.36 0.336%

Redo this arithmetic before you copy the thresholds. Suppose you keep 14.4 on a 7-day window. That pages only after you lose about 8.6% of the budget in an hour, which is four times later than intended. At 99.99%, 0.144% errors can come from a few failed requests on a low-traffic API, so the alert becomes noisy. At 99%, an outage that fails 10% of requests never fires the page.

Why are threshold alerts on raw error rates not enough, and why use multiple windows?

never crosses a 14.4% threshold and never pages, even though it exhausts a 30-day budget in three days. A fixed threshold ties the alert to one error rate, while the real question is how fast the budget is being spent.

Switching to burn rate fixes the scale, but a single window still fails. Take a 99.9% SLO, a one-hour outage at 36× burn starting 10:00, and a three-minute spike at 100× around 13:00. Each character below is 10 minutes from 10:00 to 13:10, and # means the condition is above 14.4×:

  • errors: ######............!.
  • 5m > 14.4: ######............#.
  • 1h > 14.4: ...#######..........
  • 1h AND 5m: ...###..............

The 1h window fires at about 10:24 and stays true until about 11:36, 36 minutes after recovery, because the outage is still in its average. The 5m window catches the outage within two minutes but also pages on the spike. That spike consumed under 0.5% of the budget.

Requiring both conditions keeps the long window's evidence of real budget loss and uses the short window, about 1/12 the length, as a reset. The alert clears around 11:03. The standard pairs are 1h/5m, 6h/30m, 1d/2h and 3d/6h.

Rule Fires on outage Clears after recovery Pages on spike
1h only ~24 min ~36 min late No
5m only ~2 min ~3 min Yes
1h AND 5m ~24 min ~3 min No

The long window still sets detection time. A lower threshold or a shorter window detects faster but pages on noise. A higher threshold or a longer window cuts false pages but misses slow burns. No setting gets both missed incidents and false pages to zero, so choose each pair based on how much budget you can afford to lose before someone acts.

          +----------------------------------+
          | API error ratio (errors/requests)|
          +----------------------------------+
               |                        |
               v                        v
     +--------------------+   +--------------------+
     | long window: 1h    |   | short window: 5m   |
     | error ratio / 0.001|   | error ratio / 0.001|
     | = burn rate        |   | = burn rate        |
     +--------------------+   +--------------------+
               |                        |
               v                        v
     +--------------------+   +--------------------+
     | burn > 14.4?       |   | burn > 14.4?       |
     | evidence of real   |   | still burning now; |
     | budget loss; alone:|   | alone: flaps on    |
     | slow to clear      |   | brief spikes       |
     +--------------------+   +--------------------+
               |                        |
               +------------+-----------+
                            v
                 +---------------------+
                 |         AND         |
                 +---------------------+
                            |
                            v
                 +---------------------+
                 | page: fires ~24 min |
                 | in; clears ~3 min   |
                 | after; no spike page|
                 +---------------------+

For a 99.9% SLO (error budget 0.001), each multi-window rule computes burn rate over a long window and a short window about 1/12 its length, and pages only when both exceed the threshold; the same pattern applies to the 1h/5m, 6h/30m, 1d/2h and 3d/6h pairs.

How do you implement burn rate recording rules and alerting rules in Prometheus?

Create one recording rule per window and follow the level:metric:operations naming convention, for example api:slo_errors_per_request:ratio_rate5m through ratio_rate3d. With consistent names, every alert expression looks alike, and a typo in one window is easy to spot. Compute each window directly as sum(rate(errors[w])) / sum(rate(total[w])). Do not average shorter ratios to get a longer window. An average of ratios gives a quiet hour the same weight as a busy one, which understates the error rate in the windows where it matters most.

Before you compute the 1d and 3d windows, aggregate away high-cardinality labels such as pod, instance and path. Keep only the labels you route on, such as service and SLO name. A rate() over three days touches every sample in every series, so dropping labels first is the main way to control evaluation cost.

Write the threshold as the burn-rate factor multiplied by the budget, as a literal constant. For a 99.9% SLO the budget is 0.001, so the page threshold is 14.4 * 0.001. Writing it this way keeps the factor visible to reviewers. Each alert requires both windows of its pair to exceed the threshold:

  • Page on 1h and 5m above 14.4, and on 6h and 30m above 6.
  • Open a ticket on 1d and 2h above 3, and on 3d and 6h above 1.

Set for to zero or a minute or two. The short window already filters out noise, and a longer for only delays detection. Add labels for slo, severity and window, and add an annotation that shows the current burn rate.

Burn-rate alerts fail silently when data is missing. A ratio with no samples returns nothing, and nothing never crosses a threshold. Scrape gaps and counter resets that rate() misreads can hide a real burn in the same way. Add separate alerts that use absent() on the request counter and fire on up == 0 for the scrape targets, so a missing signal pages someone instead of reading as healthy.

groups:
  - name: api-slo-recording
    rules:
      # One rule per window. Repeat this shape for 30m, 2h, 6h, 1d and 3d.
      # Grouping by service only drops pod/instance/path before the long windows.
      - record: api:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (service) (rate(http_requests_total{job="api",code=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total{job="api"}[5m]))
      - record: api:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (service) (rate(http_requests_total{job="api",code=~"5.."}[1h]))
          /
          sum by (service) (rate(http_requests_total{job="api"}[1h]))
  - name: api-slo-alerts
    rules:
      # Page: both the long and short window must exceed 14.4x the 0.1% budget.
      - alert: APIErrorBudgetBurnFast
        expr: |
          api:slo_errors_per_request:ratio_rate1h > (14.4 * 0.001)
          and
          api:slo_errors_per_request:ratio_rate5m > (14.4 * 0.001)
        for: 1m
        labels:
          severity: page
          slo: api-availability
          window: 1h/5m
        annotations:
          summary: "{{ $labels.service }} 1h burn rate is {{ $value | humanize }} error ratio (threshold 0.0144)"
      # Missing data never crosses a threshold, so alert on it directly.
      - alert: APIRequestMetricAbsent
        expr: absent(http_requests_total{job="api"})
        for: 10m
        labels:
          severity: ticket
          slo: api-availability
      - alert: APIScrapeTargetDown
        expr: up{job="api"} == 0
        for: 5m
        labels:
          severity: ticket
          slo: api-availability

Prometheus rules showing two of the per-window recording rules, the 1h/5m page alert with its 14.4 × 0.001 threshold written as a constant, and companion alerts for absent request metrics and down scrape targets.

How should Alertmanager route page and ticket burn rate alerts?

The alerting rules from the previous section should attach three labels to every burn rate alert: severity (page or ticket), service, and slo. Alertmanager uses these labels for routing, grouping, and inhibition, so set them consistently across every SLO.

Route each severity to its own receiver

Under the top-level route, add two child routes that match on severity. Send page to the on-call paging receiver and ticket to a queue that someone reviews during working hours. A slow burn that leaves days of budget does not need anyone woken at 3 a.m. Separating the receivers enforces that in configuration, so it does not depend on each responder's judgement.

Group by service and SLO

Set group_by to alertname, service, and slo. One SLO breach can make both the fast and slow windows fire, and each one can fire per instance or per route label. Grouping folds these into a single notification per objective, and that notification lists every member alert.

Inhibit tickets while a page is firing

Add an inhibition rule with these parts:

  • Source: alerts with severity="page"
  • Target: alerts with severity="ticket"
  • Equal: service and slo

The equal list matters. Without it, a page on one API would mute unrelated tickets on others. With it, the on-call engineer gets one alert about the burn, and no ticket is opened for the same problem.

Tune repeat intervals per severity

For pages, use a short repeat_interval, such as an hour, so an unacknowledged burn keeps getting attention. For tickets, use a long one, such as a day, so a slow burn does not refile the same ticket over and over.

Together these settings mean each incident reaches each person once, through the right channel. Interruptions per incident go down, and that is where most of the engineering time is saved.

# alertmanager.yml: route, group and inhibit SLO burn rate alerts
route:
  receiver: default
  # One notification per objective, not per window or instance
  group_by: ['alertname', 'service', 'slo']
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      # Keep re-notifying until someone acts on a fast burn
      repeat_interval: 1h
    - matchers: ['severity="ticket"']
      receiver: ticket-queue
      # Slow burns: avoid refiling the same ticket all day
      repeat_interval: 24h

receivers:
  - name: default
  - name: oncall-pager
    webhook_configs:
      - url: ''  # placeholder
  - name: ticket-queue
    webhook_configs:
      - url: ''  # placeholder

inhibit_rules:
  # While a page fires, mute the ticket for the same service and SLO only
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: ['service', 'slo']

An Alertmanager config that routes page and ticket burn rate alerts to separate receivers, groups by alertname, service and SLO, sets a repeat interval per severity, and inhibits tickets while a matching page is firing.

How do you test SLO alert rules with promtool before relying on them?

Burn rate rules are easy to get subtly wrong. A window that doesn't match its threshold, or a recording rule name with a typo, produces an alert that never fires. promtool test rules catches these before production does. A test file declares synthetic input series, an evaluation interval, and the expected alerts at specific eval_time values. Write at least three scenarios:

  • Sharp outage: jump the error ratio to 50% at a fixed time. Assert that the fast page fires within minutes and that nothing fires before the jump.
  • Slow leak: hold errors just above the slow-ticket burn rate for days. Assert that the ticket fires and the page never does.
  • Brief spike: add a few minutes of errors, then recover. Assert that the short window stops the alert from firing, or resolves it quickly.

Asserting on evaluation time matters because it encodes the detection latency you designed for. If someone later changes a window, the test fails.

Unit tests prove the logic. Back-testing proves the thresholds against your real traffic. Run the alert expressions as range queries over periods that include past incidents and quiet weeks. Record when each alert would have fired compared with when the incident started, and count how many pages fell outside any incident. Recording rules only exist from the time they were deployed. For history, either inline the raw expressions or backfill the recorded series first.

Development cost grows quickly. The rule count is roughly services × SLIs × windows. Generating rules from a template or an SLO spec keeps them consistent and makes review a diff of intent. The trade-off is that you now own a generator. Hand-written rules avoid that tooling, but they drift between services and are harder to review.

When should you avoid standard burn rate alerts, and what are the alternatives?

Burn rate assumes enough traffic for an error ratio to mean something. Take a 99.9% SLO on an endpoint serving about eight requests per five minutes (illustrative estimate). One failure in the short window is a 12.5% error ratio, a burn rate of 125, and the 14.4 page fires on a single bad request. Spiky batch clients cause the same problem: a burst of retries against one failing dependency can trip both windows within minutes.

The usual fixes each trade something:

  • Minimum request count in the alert expression suppresses noise, but it also hides a total outage during quiet hours, because zero requests never reach the threshold.
  • Synthetic probe traffic guarantees a baseline volume, but it only exercises the paths the probe calls.
  • Longer windows smooth the ratio and slow detection.
  • Grouping endpoints under one SLO adds volume, but a broken low-traffic endpoint can disappear inside a healthy aggregate.

Rolling windows also measure how fast the budget is burning, not how much of it remains. Release-freeze decisions need a separate budget-remaining query over the full SLO period and a dashboard built on that query.

Alternative Fits when Weakness
Static error-rate threshold Steady, high-volume service with a well-known normal rate Not tied to budget; tuned by guesswork
Single-window burn rate Moderate traffic; simplicity matters more than precision Slow reset or noisy, depending on window
Black-box probe alerting Low or bursty traffic; "is it up" is the key question Misses partial and client-specific failures

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to slo burn rate alerts prometheus, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.