Microservices Service Ownership: On-Call, SLOs and Runbooks per Team

What microservices service ownership means and who it is for

In microservices service ownership, the team that ships a service also holds its full operational contract. That contract has three parts:

• A named on-call rotation that gets paged when the service misbehaves. It is a rotation, not a single person, and it is not a shared operations queue.
• SLOs the team chose and can influence. The targets reflect what users of the service need, and the team controls the code, deploys and capacity that move those numbers.
• Current runbooks for the failures the service is known to have, written by the people who will follow them at 3 a.m.

Compare this with ownership that stops at the code. In that model a team merges and deploys, and a separate operations group handles alerts it did not design for software it did not write. Every incident starts with a handoff, and the responders learn the system while it is down. The developers rarely feel the cost of a noisy alert or a fragile retry loop, so those problems tend to stay.

Organizations adopt full ownership for two reasons. Fixes come faster when the responder knows the code. Reliability also gets an accountable owner, so missing an SLO becomes a prioritization input for a specific backlog.

The model fits backend teams that run their own services. It also fits platform engineers who give those teams shared alerting, paging and SLO tooling.

It applies when three conditions hold:

• Several teams contribute to the system.
• Services deploy independently.
• Each team has enough people to staff a sustainable rotation.

If those conditions are missing, see the alternatives in the last section.

Recording an owner for every service

Ownership has to live somewhere a machine can read it, next to the code or in a service catalog, rather than in a wiki page or someone's memory. Assign each service to exactly one team, never to an individual. People change roles, go on leave and leave the company. Teams persist, and a team can absorb those changes without the service becoming orphaned.

A useful ownership record holds more than a team name. At minimum it should include:

• Owning team: a single identifier that matches an entry in your team registry.
• Escalation contacts: a team channel and a secondary escalation path, such as an engineering manager alias.
• On-call schedule reference: the identifier your paging system uses, so alerts can be routed without a lookup table maintained by hand.
• SLO definition location: a path or reference to where the objectives are declared.
• Runbook location: a path to the runbooks, preferably in the same repository.

Treat a service without a valid owner as a defect. File it, give it a severity and track it like any other bug. Otherwise unowned services pile up quietly until one of them pages nobody at 3 a.m.

Ownership metadata drifts most after reorgs, when teams are renamed, merged or dissolved. To catch this, validate it in CI. The build should fail when a required field is missing or when the owner names a team that no longer exists in the registry. Run the same check against alert-routing configuration so that every alert resolves to a live schedule. The example YAML file and validation script below show both checks.

Copy#!/usr/bin/env python3
# CI check: fail the build when ownership metadata is incomplete or stale.
# Usage: check_ownership.py ownership.yaml teams.yaml
# Example ownership.yaml (one file per service repo):
#   owner: team-payments                 # exactly one team, never a person
#   escalation: {channel: "#payments-oncall", secondary: payments-em@example.com}
#   oncall_schedule: payments-primary    # ID used by the paging system
#   slo: slo/checkout-api.yaml
#   runbooks: docs/runbooks/
import os, sys
import yaml  # PyYAML

REQUIRED = ["owner", "escalation", "oncall_schedule", "slo", "runbooks"]

def main(ownership_path, registry_path):
    with open(ownership_path) as f:
        rec = yaml.safe_load(f) or {}
    with open(registry_path) as f:
        # Team registry: {teams: {team-id: {schedules: [schedule-id, ...]}}}
        teams = (yaml.safe_load(f) or {}).get("teams") or {}

    errors = [f"missing field: {k}" for k in REQUIRED if not rec.get(k)]
    owner, sched = rec.get("owner"), rec.get("oncall_schedule")
    if owner and not isinstance(owner, str):
        errors.append("owner must be a single team id, not a list")
    elif owner and owner not in teams:
        errors.append(f"unknown team '{owner}' (renamed or dissolved in a reorg?)")
    elif owner and sched not in (teams[owner] or {}).get("schedules", []):
        errors.append(f"schedule '{sched}' is not a live schedule of '{owner}'")
    esc = rec.get("escalation") or {}
    if not (esc.get("channel") and esc.get("secondary")):
        errors.append("escalation needs both a channel and a secondary path")
    for key in ("slo", "runbooks"):  # references must point at real files
        if rec.get(key) and not os.path.exists(rec[key]):
            errors.append(f"{key} path does not exist: {rec[key]}")

    for e in errors:
        print(f"ownership check failed: {e}", file=sys.stderr)
    sys.exit(1 if errors else 0)

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

A CI script, with a sample ownership.yaml in its header comment, that fails the build when required ownership fields are missing, the owner is not a single team in the registry, the on-call schedule does not belong to that team, or the SLO and runbook paths do not exist.

Mapping services to on-call rotations and routing pages

Every alert should carry a service label set at the source, in the metric, log rule or synthetic check. The routing layer resolves that label through the catalog to a team, then to that team's schedule. A shared "all-alerts" channel spreads responsibility so thin that nobody acts. Catalog-driven routing means a reorg changes one catalog entry instead of dozens of alert rules.

alert -> service label -> catalog lookup -> owning team -> team rotation -> page + runbook link

The page should include the runbook URL from the same catalog entry, so the responder never has to search for it at 3 a.m.

One rotation, several services

Give each team one rotation covering all its services, not one schedule per service. Per-service schedules fragment people: an engineer on three rotations is effectively always on call, and small schedules leave no slack for leave or illness. Merging rotations costs some depth, because the responder may not know every service equally well. Runbooks and a clear escalation path to the service's primary maintainers cover that gap. Merge services that share dependencies, deploy together or fail together. Keep services separate when their failure modes require different expertise and the team is large enough to staff both.

Minimum rotation size

Illustrative estimate, assuming weekly shifts with a primary and a secondary: four people means primary duty one week in four and secondary another week in four. That is half of all weeks carrying some on-call load. Below that, burnout is likely. If a team cannot staff a sustainable rotation, merge it with an adjacent team's rotation rather than shrink the shift length.

The routing example that follows matches on the team label resolved from the catalog.

Copy# Alertmanager routing keyed on the `team` label.
# The team label and runbook_url annotation are written into each alert rule
# from the service catalog entry, so a reorg updates the catalog, not this file.
route:
  # Unmatched alerts land here so missing labels get fixed, not ignored.
  receiver: catalog-gap
  group_by: ['team', 'service', 'alertname']
  routes:
    - matchers: ['team="payments"']
      receiver: payments-oncall
    - matchers: ['team="checkout"']
      receiver: checkout-oncall

receivers:
  # One receiver per team rotation, covering all of that team's services.
  - name: payments-oncall
    pagerduty_configs:
      - routing_key: '<payments-integration-key>'
        details:
          service: '{{ .CommonLabels.service }}'
          # Runbook link from the same catalog entry, shown on the page.
          runbook: '{{ .CommonAnnotations.runbook_url }}'
  - name: checkout-oncall
    pagerduty_configs:
      - routing_key: '<checkout-integration-key>'
        details:
          service: '{{ .CommonLabels.service }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'
  # Owned by the team that maintains the catalog and alerting pipeline.
  - name: catalog-gap
    pagerduty_configs:
      - routing_key: '<platform-integration-key>'
        details:
          problem: 'Alert has no team label or no matching route'
          service: '{{ .CommonLabels.service }}'

An Alertmanager route tree that sends each alert to its owning team's rotation based on the catalog-derived team label, attaches the runbook URL to the page, and sends unlabeled alerts to a fallback receiver so labeling gaps get fixed.

Choosing team SLOs and paging on error-budget burn

Routing only helps if the pages are worth sending. Each team should define SLIs at its own service boundary, measured where callers experience the service. Three SLIs cover most services. Availability is the ratio of good responses to valid requests. Latency is the share of requests faster than a threshold. Correctness is the share of responses that pass a validation check. Set targets from several weeks of observed data, slightly below what the service already achieves. A target the service has never met pages constantly and gets ignored. Do not set SLOs on dependencies the team cannot change, because those pages will reach people who can only wait.

Illustrative estimate (assumed numbers, not a benchmark). Assume an availability SLO of 99.9% of requests over 30 days (720 h). The error budget is 1 − 0.999 = 0.1%.

• The burn rate is the observed error ratio divided by the budget.
• The budget consumed equals burn rate × window ÷ 720 h.
• Paging at a burn rate of 14.4 over 1 h consumes 14.4 × 1 ÷ 720 = 2% of the budget. That burn rate corresponds to an error ratio of 14.4 × 0.1% = 1.44%.
• The page fires only if both the 1 h window and a 5 m window exceed 14.4. The short window lets the alert clear soon after recovery.
• A burn rate of 1 sustained over 3 days consumes 10% of the budget. That opens a ticket rather than a page.

Review every alert and confirm it maps to an SLO or to a written exception, such as an expiring TLS certificate.

Burn-rate alerts are harder to tune and to explain to new on-call engineers. A service with only a few requests per hour can cross 14.4× because of a single failure. For those services, use synthetic probes to generate traffic or measure the SLI over a longer window.

Copy# SLO: 99.9% of checkout requests succeed (non-5xx) over 30 days.
# Error budget = 0.1%; budget threshold = burn rate x 0.001.
groups:
  - name: checkout-slo
    rules:
      # Error ratio over each window used below.
      - record: job:slo_errors:ratio_rate5m
        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))
      - record: job:slo_errors:ratio_rate1h
        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[1h])) / sum(rate(http_requests_total{job="checkout"}[1h]))
      - record: job:slo_errors:ratio_rate6h
        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[6h])) / sum(rate(http_requests_total{job="checkout"}[6h]))
      - record: job:slo_errors:ratio_rate3d
        expr: sum(rate(http_requests_total{job="checkout",code=~"5.."}[3d])) / sum(rate(http_requests_total{job="checkout"}[3d]))

      # Fast burn: 14.4 x 0.001 = 1.44% errors. 1h at this rate spends 2% of budget.
      # The 5m window must agree, so the page clears soon after recovery.
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          job:slo_errors:ratio_rate1h > (14.4 * 0.001)
          and
          job:slo_errors:ratio_rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout burning error budget at >14.4x (2% of 30d budget per hour)"

      # Slow burn: burn rate 1 for 3d spends 10% of budget. Ticket, not page.
      - alert: CheckoutErrorBudgetSlowBurn
        expr: |
          job:slo_errors:ratio_rate3d > (1 * 0.001)
          and
          job:slo_errors:ratio_rate6h > (1 * 0.001)
        labels:
          severity: ticket
        annotations:
          summary: "Checkout consuming error budget at >=1x over 3 days"

Prometheus recording and alerting rules for a 99.9% availability SLO. A 14.4x burn on both the 1h and 5m windows pages someone. A sustained 1x burn over 3d, confirmed by the 6h window, opens a ticket.

SLOs across dependency chains and error-budget policy

A service cannot reliably beat the availability of the dependencies it needs to answer a request. If every call to checkout requires auth, inventory and pricing, each of their failures becomes a checkout failure.

Composing availability

For serial, hard dependencies with independent failures:

A_total ≈ A_self × A_dep1 × A_dep2 × … × A_depN

Illustrative estimate (assumed numbers, not a benchmark): assume checkout's own code is 99.95% available and three critical dependencies are each 99.9%.

0.9995 × 0.999 × 0.999 × 0.999 ≈ 0.9965, or about 99.65%.

A 99.9% target for checkout is out of reach. Now assume pricing is served from a cache when it fails, which takes it off the critical path:

0.9995 × 0.999 × 0.999 ≈ 0.9975

Caching and graceful degradation remove factors from the product. Retries help only when failures are transient and independent. During a sustained outage, retries add load and fix nothing.

Agreements between teams

Write down what each service expects from its upstreams: the SLO it relies on, the timeout budget, and the fallback it uses on failure. This exposes mismatches before an incident does.

Agree on an error-budget policy with product owners before you need it. For example, when the budget is exhausted, risky launches pause and reliability work takes priority. A policy without product buy-in gets overridden at the first deadline. Targets that are too high consume engineering time for gains users never notice. Targets that are too low, or never enforced, mean nothing.

Strict team boundaries also slow incidents that cross services. Without a shared incident-command practice, such as a single incident lead and a common channel, each team investigates only its own piece.

Writing runbooks that stay useful

Every alert should carry a link to its runbook, so the engineer paged at 3 a.m. lands on instructions instead of searching a wiki. A useful runbook answers six questions:

• Alert name and meaning: what condition fired, in plain language.
• User impact: who is affected and how badly, which sets the urgency.
• First diagnostic steps: the specific dashboards, logs or queries to check first.
• Safe mitigations: rollback, scaling out, or disabling a feature flag, with the conditions under which each is safe.
• Escalation path: who to page next, including owners of upstream dependencies.
• Known false positives: causes that trigger the alert without real impact.

Keeping runbooks alive

Store runbooks in the service repository, next to the alert definitions. A pull request that changes an alert threshold or service behavior should update the runbook in the same diff, and reviewers should check that it does. Run the runbooks during game days and onboarding. A new engineer following the steps on a staging incident will quickly find missing context. After each incident, add a follow-up item to correct whatever the runbook got wrong.

A stale runbook is a real hazard. A rollback step written before a schema migration, for example, can turn a partial outage into data loss. Watch for the opposite failure too. If every alert must have a runbook, teams can fill the requirement with boilerplate nobody reads. When an alert does not justify a runbook, consider deleting the alert.

The template below follows the six questions above.

Copy# Runbook: CheckoutHighErrorRate

Owner: team-payments | Alert defined in: alerts/checkout.rules.yml
Last verified: 2026-10-01 (game day) | Update this file in any PR that changes the alert

## What fired
5xx ratio on checkout-api above 2% for 10 minutes, measured at the load balancer.

## User impact
Customers cannot complete purchases. Treat as urgent during business hours and peak sales.

## First diagnostic steps
1. Open the "checkout-api / golden signals" dashboard. Is the error rate rising or flat?
2. Check whether a deploy happened in the last 60 minutes (deploy history for checkout-api).
3. Filter logs: service=checkout-api level=error, group by error type.
4. Check upstream health: payments-gateway and inventory-svc dashboards.

## Safe mitigations
- Roll back the last deploy: ONLY if it contained no schema migration. If it did, escalate instead.
- Scale out: safe if CPU or saturation is high and the database is not the bottleneck.
- Disable flag `checkout.new-pricing`: safe at any time; falls back to the old pricing path.

## Escalation
1. Secondary on-call for team-payments (paging rotation)
2. payments-gateway owners if errors point upstream
3. Incident commander if impact lasts more than 30 minutes

## Known false positives
- Synthetic load tests in production without the test header set.
- Upstream maintenance window for payments-gateway (check the shared change calendar).

## Follow-ups
After an incident, record what this runbook got wrong and fix it in the follow-up PR.

A Markdown runbook template that answers the six questions with concrete placeholders, states when each mitigation is safe, and records an owner, the alert definition path and a last-verified date so the runbook gets reviewed alongside the alert.

Alternatives, cost and when to avoid full ownership

When to avoid or delay full ownership

• Small teams with many services. A team of four owning twelve services gets frequent shifts with little depth. Consider merging services or sharing a rotation first.
• Poor alert hygiene. If alerts are noisy, routing them to developers only moves the pain. Clean up the alerts before moving the pages.
• Low-traffic services. These have too few requests for a meaningful error rate. Use synthetic checks instead of SLOs.

Cost against time saved

Ownership costs on-call hours, SLO and runbook upkeep, and a real risk of burnout and attrition. It saves time through faster routing to the right team, fewer escalations and fewer repeat incidents. To compare the two, apply this formula to your own figures:

net hours = (escalation hours avoided + repeat-incident hours avoided) − (on-call hours + upkeep hours)

Platform's role

The platform team should provide paved-road tooling: the service catalog, SLO templates, page routing, dashboards and runbook scaffolding. It should not answer pages for other teams' services, or it becomes a de facto ops team. Without shared tooling, the opposite problem appears: every team builds its own inconsistent, duplicated practice.

Health signals

Track these as signals, not targets:

• Pages per shift
• Share of alerts that have an owner and a runbook
• Time to engage the correct team
• Recurring-incident rate

Turning them into targets invites gaming. For example, if the goal is fewer pages per shift, teams can meet it by silencing useful alerts.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservices service ownership, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.