Argo Rollouts Canary Analysis with Prometheus: Automated Rollback

What Argo Rollouts canary analysis is and how a Rollout progresses

Progressive delivery means shipping a new version to a small share of production traffic, measuring how it behaves, and widening exposure only while the evidence stays good. Automated rollback is the other half: when the evidence turns bad, the system reverts without waiting for a human.

Argo Rollouts provides both through a Rollout custom resource that replaces a Deployment. The pod template and selector look the same, but spec.strategy.canary.steps defines an ordered plan:

• setWeight sends a percentage of traffic to the canary.
• pause holds for a fixed duration, or indefinitely until someone promotes the rollout.
• analysis blocks progression until a metric check passes.

The controller manages two ReplicaSets, stable and canary, and moves through the steps until the canary becomes the new stable. If no traffic router is configured, the controller approximates the weight by adjusting replica counts.

Analysis uses two resources. An AnalysisTemplate is the reusable definition: Prometheus queries, success conditions and failure limits. For each check, the controller creates an AnalysisRun from the template, and that run queries Prometheus and records the results.

You can attach analysis in two ways:

• Inline step analysis is a gate. The rollout waits at that step until the run finishes.
• Background analysis, set under strategy.canary.analysis, runs continuously alongside the steps. It catches regressions that appear between gates.

If an AnalysisRun fails, the controller aborts the rollout. It routes traffic back to the stable ReplicaSet, scales the canary down and marks the Rollout Degraded. The bad version stops receiving traffic before anyone has to page in.

CopyapiVersion: argoproj.io/v1alpha1
kind: Rollout                      # replaces the Deployment
metadata:
  name: checkout
spec:
  replicas: 10
  selector:
    matchLabels:
      app: checkout
  template:                        # same pod template a Deployment would use
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.4.0
  strategy:
    canary:
      # Background analysis: runs alongside the steps from step 1 onward
      analysis:
        startingStep: 1
        templates:
          - templateName: error-rate        # AnalysisTemplate with the Prometheus query
        args:
          - name: service
            value: checkout
      steps:
        - setWeight: 10                     # ~10% of traffic (via replica count if no router)
        - pause: {duration: 5m}             # let metrics accumulate
        - analysis:                         # inline gate: blocks until this AnalysisRun finishes
            templates:
              - templateName: error-rate
            args:
              - name: service
                value: checkout
        - setWeight: 50
        - pause: {}                         # wait indefinitely for manual promotion
        # after the last step, the canary becomes the new stable ReplicaSet

A minimal Rollout that shifts traffic from 10% to 50% with an inline analysis gate and continuous background analysis, both of which abort the rollout and return traffic to stable if their AnalysisRun fails.

Who should use it and when it fits

Teams adopt metric-gated canaries for two reasons. First, a bad release reaches only a slice of users for a bounded window instead of the whole fleet. Second, nobody has to sit in front of a dashboard after every deploy deciding whether a bump in errors is noise or a regression. The controller makes that call the same way every time, including at 2 a.m. and on the fifteenth deploy of the day.

The teams that benefit most share three traits:

• Frequent deploys. If you ship several times a day, manual watching doesn't scale and people stop doing it carefully. Automating the judgment pays off in proportion to how often you release.
• Enough request volume. A canary at 10% weight needs to see enough requests during each analysis interval for an error rate to mean something. A service handling a few requests per minute will produce ratios that swing wildly on a single failure.
• Existing Prometheus instrumentation. If your services already export request counts and latency histograms with consistent labels, most of the work is done. If they don't, instrument them first.

The best fit is a stateless HTTP or gRPC service where success and failure are visible per request: status codes, latency, timeouts. Rolling back such a service is safe because the old pods can resume serving immediately. Nothing has to be migrated back.

There is a catch that shapes the rest of this tutorial. An automated rollback is only as trustworthy as the queries behind it. A query that mixes stable and canary traffic, or returns no data when it should fail, will approve a broken release with confidence. That is why the metrics come before the YAML.

Designing the metrics first: isolating canary traffic and choosing signals

come first, before any YAML. Start by making sure the canary is addressable in Prometheus. Argo Rollouts labels each ReplicaSet's pods with rollouts-pod-template-hash. If your scrape config copies pod labels onto series, it appears as rollouts_pod_template_hash, and you can pass the canary's hash into the template as an argument. A version label set in the pod template works the same way. Without one of these, every query measures the blended fleet, and at a 5% canary weight a doubled error rate barely moves the aggregate.

Next, use ratios instead of raw counts. A canary at low weight serves few requests, so "fewer than 50 errors" will almost always pass. The error rate normalizes for traffic share: sum(rate(http_requests_total{rollouts_pod_template_hash="$HASH",code=~"5.."}[5m])) / sum(rate(http_requests_total{rollouts_pod_template_hash="$HASH"}[5m])). For latency, aggregate the histogram buckets before you compute the quantile: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{rollouts_pod_template_hash="$HASH"}[5m]))). Averaging per-pod percentiles is not statistically valid.

Finally, decide what "no data" means. If the canary received no requests, the ratio is 0/0, and PromQL returns NaN. If no series exist, it returns an empty result. Add a volume floor such as and on() sum(increase(http_requests_total{...}[5m])) > 200 so that low-traffic intervals return nothing. Then write the success condition so that it requires a non-empty, numeric result. When neither the success condition nor the failure condition matches, the measurement counts as inconclusive. That outcome pauses the rollout for a human, which is safer than letting missing data pass or fail by accident.

CopyapiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-rate-and-latency
spec:
  args:
    # Supplied by the Rollout, e.g. valueFrom.podTemplateHashValue: Latest
    - name: canary-hash
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      # Empty or NaN results match neither condition, so the measurement is Inconclusive
      successCondition: len(result) > 0 && !isNaN(result[0]) && result[0] < 0.01
      failureCondition: len(result) > 0 && !isNaN(result[0]) && result[0] >= 0.05
      provider:
        prometheus:
          address:
          # Only canary pods are counted. The 'and on()' floor drops the result when the
          # canary served <= 200 requests in 5m, which also prevents a 0/0 NaN.
          query: |
            (
              sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[5m]))
              /
              sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m]))
            )
            and on() (sum(increase(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m])) > 200)
    - name: p99-latency
      interval: 1m
      count: 5
      successCondition: len(result) > 0 && !isNaN(result[0]) && result[0] < 0.5
      failureCondition: len(result) > 0 && !isNaN(result[0]) && result[0] >= 1.0
      provider:
        prometheus:
          address:
          # Sum the buckets across canary pods by 'le' first, then compute the quantile once
          query: |
            histogram_quantile(0.99,
              sum by (le) (rate(http_request_duration_seconds_bucket{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m]))
            )
            and on() (sum(increase(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m])) > 200)

This AnalysisTemplate scopes its PromQL to the canary's pod-template-hash and applies a 200-request volume floor. Its conditions require a non-empty, non-NaN value, so an empty or low-traffic result is marked Inconclusive rather than passing or failing.

Writing a reusable Prometheus analysis template

Start with the template's args block, because every other field references it. Declare service-name, canary-hash, and error-threshold as arguments. Inside the query, use {{args.service-name}} and {{args.canary-hash}}. Filter on the pod-template-hash label so the query only sees canary pods. The same file then works for every service, and teams tune thresholds without forking it.

Under provider.prometheus, set address to the in-cluster Prometheus URL. Write a query that returns a single scalar, such as the canary's 5xx ratio over the last few minutes. successCondition and failureCondition are expressions evaluated against result[0]. If a value falls between them, or the query returns no data, the measurement is inconclusive rather than failed.

interval and count decide how long evidence accumulates. failureLimit is how many failed measurements the run tolerates. The run fails once failures exceed that number, which triggers the abort. inconclusiveLimit works the same way for ambiguous measurements, and exceeding it pauses the rollout instead of aborting it.

In the Rollout's analysis step, pass literal values for name and threshold. For the hash, use valueFrom.podTemplateHashValue: Latest, so the hash always tracks the current canary revision.

CopyapiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  args:
    - name: service-name
    - name: canary-hash
    - name: error-threshold
      value: "0.05"      # default; callers can override per service
  metrics:
    - name: canary-5xx-ratio
      interval: 1m          # one measurement per minute
      count: 5              # five measurements, then the run completes
      failureLimit: 1       # fails once failures exceed 1
      inconclusiveLimit: 1  # pauses once inconclusive results exceed 1
      successCondition: result[0] < {{args.error-threshold}}
      failureCondition: result[0] >= {{args.error-threshold}}
      provider:
        prometheus:
          address:
          # Assumes scrape relabeling maps the pod label
          # rollouts-pod-template-hash to rollouts_pod_template_hash.
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",
              rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[3m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}",
              rollouts_pod_template_hash="{{args.canary-hash}}"}[3m]))
---
# Excerpt from a Rollout's spec.strategy.canary.steps
- setWeight: 20
- analysis:
    templates:
      - templateName: error-rate
    args:
      - name: service-name
        value: checkout
      - name: error-threshold
        value: "0.02"     # stricter threshold for this service
      - name: canary-hash
        valueFrom:
          podTemplateHashValue: Latest  # tracks the current canary revision

A reusable AnalysisTemplate that measures the canary's 5xx ratio and the Rollout step that supplies the service name, a per-service threshold and the canary pod-template hash as arguments.

Traffic splitting and what automated rollback actually does

Without a traffic router, Argo Rollouts approximates setWeight by scaling the canary ReplicaSet relative to the stable one. The Service selects pods from both, so the split depends on pod counts and on how kube-proxy balances connections. With four replicas, the smallest possible step is about 25%. A requested 5% therefore exposes far more users than planned. Long-lived connections such as HTTP/2 or gRPC make the split less predictable still, because traffic is balanced per connection rather than per request. When the real ratio drifts from the requested one, any query that compares absolute request or error counts between canary and stable becomes misleading. This is why the template should compare ratios, not totals.

A supported ingress controller or service mesh splits traffic at the request level. It decouples weight from replica count, so a single canary pod can receive exactly 5%.

The abort path

1. A measurement exceeds failureLimit, and the AnalysisRun is marked Failed.
2. The controller aborts the Rollout and stops further steps.
3. Traffic weight returns to 100% stable. The canary ReplicaSet is then scaled down. With a traffic router, this happens after abortScaleDownDelaySeconds.
4. The Rollout reports Degraded. The spec still points at the bad image, so nothing is actually reverted in Git or the manifest. You resolve this in one of two ways: push a spec change, either a fix or the previous image, or run kubectl argo rollouts retry rollout <name> to attempt the same revision again.

Testing the rollback path before you trust it

That is the end state you want to see in a drill, so produce it on purpose in a staging cluster before production depends on it.

Force a failure

Ship an image that returns 5xx on a fraction of requests, or add an environment flag that makes a known endpoint error, so the failure path is deterministic. Run load against the service; without canary traffic, the analysis has nothing to measure. In one terminal, follow kubectl argo rollouts get rollout <name> --watch. In another, run kubectl describe analysisrun <run-name> and read each measurement's value and phase. Confirm three things:

• The failure is counted against the expected failureLimit.
• The Rollout reports Degraded and the canary weight drops to zero.
• The stable ReplicaSet serves all traffic without a restart.

Then recover once using each method: retry with the bad image still in the spec, which should fail again, and a spec update to a good image, which should progress normally.

Test missing data

Run the same procedure with no load, and again with Prometheus blocked by a NetworkPolicy or scaled to zero. These cases behave differently:

• Unreachable provider. Each failed query is recorded as an Error measurement. Once errors exceed the consecutive error limit, the run fails and the Rollout aborts.
• Zero traffic. The query may return an empty result or NaN. Depending on how your conditions are written, the measurement can error, fail, pass, or come back Inconclusive. An inconclusive result pauses the Rollout until someone acts, and it does not abort.

Make sure an idle canary never passes. One common cause is or vector(0), which can make "no data" look like "no errors."

Copy#!/usr/bin/env bash
# Rollback drill for a staging cluster. Placeholders: rollout "checkout", container "app".
set -euo pipefail
NS=staging
RO=checkout
BAD=registry.example.com/checkout:fault-5xx   # image that returns 5xx on some requests
GOOD=registry.example.com/checkout:1.4.2

# 1. Force a failure. Keep load running against the service, or there is nothing to measure.
kubectl argo rollouts set image "$RO" app="$BAD" -n "$NS"

# Terminal A: watch canary weight, step, and status (expect Degraded, canary weight 0).
#   kubectl argo rollouts get rollout "$RO" -n "$NS" --watch

# Terminal B: find the newest AnalysisRun and read each measurement's value and phase.
RUN=$(kubectl get analysisrun -n "$NS" \
  --sort-by=.metadata.creationTimestamp -o name | tail -n 1)
kubectl describe "$RUN" -n "$NS"

# 2. Recover, once per method.
# Retry with the bad image still in the spec: this should fail again.
kubectl argo rollouts retry rollout "$RO" -n "$NS"
# Update the spec to a good image: this should progress normally.
kubectl argo rollouts set image "$RO" app="$GOOD" -n "$NS"

# 3. Missing data, case A: Prometheus unreachable.
# Expect Error measurements, then a failed run and an abort once the consecutive error limit is exceeded.
kubectl scale deployment prometheus-server -n monitoring --replicas=0
kubectl argo rollouts set image "$RO" app=registry.example.com/checkout:1.4.3 -n "$NS"
# ...observe with the commands above, then restore:
kubectl scale deployment prometheus-server -n monitoring --replicas=1

# 4. Missing data, case B: zero traffic. Stop the load generator and repeat step 3's deploy.
# The run must NOT end Successful. Inconclusive pauses the Rollout without aborting it.
# Check what the metric query actually returned:
#   kubectl get "$RUN" -n "$NS" -o jsonpath='{.status.metricResults[*].measurements[*].value}'

A staging drill script that forces a canary failure, inspects the AnalysisRun, recovers through both retry and an image update, then repeats the rollout with Prometheus scaled down and with no traffic to see how missing data is handled.

When to avoid it, alternatives, and impact on cost and engineering time

Canary analysis is a poor fit for low-traffic services. At a few requests per minute, one failed call can push an error rate over the threshold. You then need longer windows, and the rollout moves only as fast as the statistics allow. Rollback also covers only what you measure. Wrong prices, corrupted writes, or a downstream consumer that breaks on a new field can pass every check and get promoted. Migrations, sent emails and published messages are not reverted, so every change must work while both versions run at the same time.

Prometheus becomes part of the deployment critical path, and an outage or scrape lag stalls releases. Static thresholds drift as traffic changes, so review them on a schedule as you would alerts. Someone has to own paused, inconclusive and aborted Rollouts. Adoption also brings a controller, new CRDs, GitOps health checks for Rollout resources, and dashboard changes.

Illustrative estimate (assumed numbers, not a benchmark). Monthly hours saved = R × W + B × D × E.

Assumed inputs:

• R = 20 releases
• W = 0.5 h of watching per release
• B = 2 bad releases
• D = 0.33 h earlier containment
• E = 3 responders

That gives 10 + 2 = 12 h/month. Assume 40 h of setup and 4 h/month of maintenance. Net savings are 8 h/month, so setup pays back in about five months.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to argo rollouts canary analysis, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.