What is an Argo Rollouts canary and how does it replace a Deployment?

Progressive delivery means releasing a new version to a small slice of production traffic, measuring how it behaves, and only then widening exposure. A Deployment cannot do this. Its rolling update only checks readiness probes, and it has no concept of pausing at 10% or of judging a version by its error rate.

Argo Rollouts adds a Rollout custom resource whose spec mirrors a Deployment's: replicas, selector, and template. It replaces the strategy field with a canary block that lists steps such as set weight, pause, and analyze. The Rollout controller, not the Deployment controller, creates and scales the ReplicaSets. During an update it keeps two of them: the stable ReplicaSet serving the current version and the canary ReplicaSet for the new one. The controller labels each ReplicaSet and its pods with rollouts-pod-template-hash. If you configure separate stable and canary Services, it injects that hash into their selectors so each Service targets exactly one version.

Migrating without downtime

You have two options:

  • workloadRef: the Rollout points at your existing Deployment and reuses its pod template, so the Deployment manifest stays the source of truth.
  • Convert the manifest: change the kind and apiVersion to the Rollout's and move the strategy into canary.

In both cases, deploy the Rollout alongside the running Deployment. Wait until the Rollout's pods are healthy, then scale the Deployment to zero. If a GitOps tool manages the Deployment, make that change in Git, or the tool will scale it back up.

Updating surrounding tooling

Anything that assumes the Deployment kind needs changes:

  • Autoscaling: point HPA scaleTargetRef at the Rollout.
  • Day-to-day commands: replace kubectl rollout with the kubectl argo rollouts plugin.
  • GitOps health checks: confirm your GitOps tool understands Rollout health status.
# Rollout that reuses an existing Deployment's pod template via workloadRef.
# Apply it alongside the running Deployment; once the Rollout's pods are
# healthy, set the Deployment's replicas to 0 (in Git, if GitOps manages it).
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 4
  selector:
    matchLabels:
      app: checkout          # must match the Deployment's pod labels
  workloadRef:               # pod template comes from this Deployment
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  strategy:
    canary:
      steps:
        - setWeight: 25        # shift 25% of pods to the new version
        - pause: {duration: 5m}
        - setWeight: 50
        - pause: {}            # wait for manual promotion
---
# The HPA now scales the Rollout instead of the Deployment.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: checkout
  minReplicas: 4
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}

A Rollout that adopts an existing Deployment's pod template through workloadRef, with canary steps and an HPA retargeted from the Deployment to the Rollout.

How does canary traffic shifting work with and without a traffic router?

A canary strategy is an ordered list of steps. setWeight moves a percentage of traffic to the new version. pause holds the rollout, either for a fixed duration or indefinitely until someone runs kubectl argo rollouts promote. An analysis step creates an AnalysisRun that queries Prometheus and blocks progress until the run succeeds or fails. A failed run aborts the rollout and shifts traffic back to stable.

How a weight becomes real traffic depends on whether a traffic router is configured.

Without a router, the controller has only one control: the replica count. Stable and canary pods sit behind the same Service, so each pod receives roughly an equal share of connections. The weight is therefore approximated by pod ratio and rounded to whole pods.

Illustrative estimate: with 4 replicas, setWeight: 10 cannot produce 0.4 of a pod. One canary pod takes about 25% of requests, which over-exposes the canary. With 40 replicas the same step is accurate, but the rollout costs a full set of extra pods.

With a router, such as a service mesh or an ingress controller, the controller writes weights into the router's routing rules. Replica count and traffic share are then independent, so 2 canary pods can serve exactly 5% of requests.

Aspect Replica-based Traffic router
Precision at 1–10% Coarse, rounded to pods Exact percentage
Extra infrastructure None Mesh or ingress controller
Header/cookie routing No Often supported
Rollback speed Scale down canary pods Router weight change 
             +--------------------+
             | Rollouts controller|
             +--+------+------+---+
     scales RS  |      |      | creates
                v      |      v
   +---------+  +-----+ | +-------------+   query  +------------+
   | stable  |  |canary| | | AnalysisRun |--------->| Prometheus |
   |   RS    |  |  RS  | | +-------------+          +------------+
   +----^----+  +--^---+ | sets weights
        |  95%     | 5%  v
        +----+-----+ +----------------+
             +-------| traffic router |<--- client requests
                     +----------------+

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 4
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:2.4.0
  strategy:
    canary:
      # Separate Services let the router address each ReplicaSet.
      stableService: checkout-stable
      canaryService: checkout-canary
      # Remove this block and weights fall back to pod ratios
      # (4 replicas: setWeight 5 becomes 1 pod, about 25%).
      trafficRouting:
        plugins:
          # Gateway API plugin; must be registered in the controller ConfigMap.
          argoproj-labs/gatewayAPI:
            httpRoute: checkout-route
            namespace: shop
      steps:
        - setWeight: 5            # router sends exactly 5% to canary
        - pause: {duration: 10m}  # let metrics accumulate
        - analysis:               # AnalysisRun queries Prometheus; failure aborts
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: {}               # wait for: kubectl argo rollouts promote checkout
        - setWeight: 50
        - pause: {duration: 10m}

A canary Rollout that uses setWeight, pause, and an analysis step, with a Gateway API traffic router so a 5% weight is applied exactly instead of being rounded to whole pods.

How do you write an AnalysisTemplate that queries Prometheus?

An AnalysisTemplate has two parts: args, which the Rollout fills in (for example a service name or the canary pod-template hash), and metrics, where each metric has a Prometheus address and a query that references args as {{args.service-name}}. The query should return one scalar for the canary only. If it aggregates over the stable pods, a broken canary carrying 5% of traffic barely moves the result.

interval sets how often the query runs, and count sets how many measurements are taken. successCondition and failureCondition are expressions evaluated against result. If a measurement matches neither, it is inconclusive. failureLimit is how many failed measurements the run tolerates before it fails and triggers rollback. inconclusiveLimit does the same for ambiguous measurements, which pause the rollout for a human to decide.

Make missing data fail

Missing data is the dangerous case. A ratio of errors to requests returns NaN when the canary served no traffic. An absent series returns an empty vector. A careless or vector(0) turns both into a perfect score. Guard in two places:

  • In the condition. Write len(result) > 0 && !isNaN(result[0]) && result[0] < 0.01. An empty or NaN result then counts as a failure instead of a pass.
  • In the PromQL. Use or vector(-1) to map absence to a sentinel. Either reject -1 explicitly in failureCondition, or leave both conditions unmatched so the measurement lands as inconclusive.

Prometheus becomes release-critical

These guards make Prometheus a release-critical dependency. A scrape gap or a restarting Prometheus now halts deployments. Recording rules make this worse because they lag raw data by their evaluation interval. Set interval well above that lag, and give the canary enough warm-up that the first measurements see real samples.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-rate
spec:
  args:
    - name: service-name
    - name: canary-hash   # Rollout passes this via valueFrom.podTemplateHashValue: Latest
  metrics:
    - name: error-rate
      interval: 2m           # keep well above scrape and recording-rule evaluation lag
      count: 5               # five measurements, then the run completes
      failureLimit: 1        # a second failed measurement aborts and rolls back
      inconclusiveLimit: 1   # a second ambiguous measurement pauses for a human
      # Pass only on a real, non-negative ratio under 1%. Empty, NaN and -1 never pass.
      successCondition: >-
        len(result) > 0 && !isNaN(result[0]) && result[0] >= 0 && result[0] < 0.01
      # NaN means 0/0: the canary received no requests, so fail it.
      # -1 (series absent) matches neither condition and counts as inconclusive.
      failureCondition: >-
        len(result) == 0 || isNaN(result[0]) || result[0] >= 0.05
      provider:
        prometheus:
          address:
          # Assumes scrape config copies the pod label to rollouts_pod_template_hash.
          # Filters to canary pods only, so stable traffic cannot dilute the ratio.
          query: |
            (
              sum(rate(http_requests_total{service="{{args.service-name}}",
                rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[2m]))
              /
              sum(rate(http_requests_total{service="{{args.service-name}}",
                rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))
            ) or vector(-1)  # a missing series becomes a -1 sentinel, never a fake 0

An AnalysisTemplate that scopes the Prometheus error-ratio query to canary pods, maps an absent series to a -1 sentinel that lands as inconclusive, and treats empty or NaN results as failures instead of passes.

Which Prometheus metrics actually catch a bad canary?

Before any of that matters, the query has to see only canary pods. A cluster-wide error rate at a 10% canary weight is roughly 90% stable traffic, so a canary that fails one request in five shows up as about a 2% overall error rate. Argo Rollouts labels each ReplicaSet's pods with rollouts-pod-template-hash. Your scrape config must relabel that pod label onto the series. Hyphens are invalid in Prometheus label names, so it typically becomes rollouts_pod_template_hash. Pass the canary hash into the template as an argument and filter on it.

Measure ratios, not counts. Use 5xx responses divided by total requests rather than raw error counts, because counts scale with traffic weight. For latency, use histogram_quantile over bucket rates rather than averages. A mean hides a slow tail that users feel at p99.

Approach Catches Weakness
Absolute threshold (e.g. error ratio < 1%) Clear breakage Ignores normal baseline; misses gradual regression
Canary vs. stable baseline (canary ≤ 1.5× stable) Relative regressions Noisy when the stable error rate is near zero; stable incidents mask canary ones

A common choice is to combine both: an absolute ceiling plus a relative check.

Illustrative estimate. This example assumes 200 req/s total, a 10% canary weight, and a 0.1% baseline error rate. Over a 1-minute window, the canary serves 1,200 requests and you expect about 1.2 errors. A healthy canary would then log three or more errors, a 0.25% rate, roughly 12% of the time by chance. Over 5 minutes the canary serves 6,000 requests, and that noise shrinks considerably. Longer windows mean fewer false rollbacks but slower releases. Low-traffic services may need longer steps or a count that requires several consecutive failures.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-ratio
spec:
  args:
    - name: canary-hash   # Rollout passes valueFrom.podTemplateHashValue: Latest
    - name: stable-hash   # Rollout passes valueFrom.podTemplateHashValue: Stable
  metrics:
    # Absolute ceiling: canary-only 5xx ratio must stay under 1%.
    - name: canary-error-ratio
      interval: 1m
      count: 5            # five measurements per analysis run
      failureLimit: 1     # tolerate one noisy measurement; fail the run on a second
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address:
          # 'or vector(0)' turns "no 5xx series yet" into 0 instead of an empty result
          query: |
            (sum(rate(http_requests_total{code=~"5..", rollouts_pod_template_hash="{{args.canary-hash}}"}[5m])) or vector(0))
            /
            sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m]))
    # Relative check: canary ratio <= 1.5x stable. clamp_min sets a 0.1% floor on the
    # stable ratio so a near-zero baseline does not turn one canary error into a failure.
    - name: canary-vs-stable
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] <= 1.5
      provider:
        prometheus:
          address:
          query: |
            ((sum(rate(http_requests_total{code=~"5..", rollouts_pod_template_hash="{{args.canary-hash}}"}[5m])) or vector(0))
              / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[5m])))
            /
            clamp_min(
              (sum(rate(http_requests_total{code=~"5..", rollouts_pod_template_hash="{{args.stable-hash}}"}[5m])) or vector(0))
                / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.stable-hash}}"}[5m])),
              0.001)

An AnalysisTemplate that filters on the relabeled rollouts_pod_template_hash label and pairs an absolute 1% canary error-ratio ceiling with a canary-versus-stable check that has a floor on the baseline, using 5-minute rate windows to reduce noise at low traffic.

Should you use background analysis or inline step analysis?

Background analysis is declared once on the canary strategy and runs continuously from a chosen step, set with startingStep, until the rollout finishes or aborts. Inline analysis is a step in the step list. It blocks progression until its AnalysisRun completes, so it gates one specific weight.

The two forms differ in what they catch. Background analysis watches for degradation at any point, including during pauses, when a slow memory leak or a creeping latency regression has time to appear. Inline analysis answers a narrower question: is this canary healthy enough to move from 20% to 50%? Because an inline step must end before the rollout continues, its metrics need a count. Background metrics usually omit count so they keep running for the whole rollout.

A practical combination is to run background analysis on error rate and saturation as a tripwire, and place inline analysis before the large weight jumps, where a stricter latency comparison justifies waiting.

Long pauses cause problems when you need to ship an urgent fix. A rollout with an hour of steps also holds the hotfix for an hour. Document the override path before you need it:

  • kubectl argo rollouts promote --full skips the remaining steps and analysis.
  • kubectl argo rollouts abort returns traffic to the stable version.

Restrict who can run these commands through RBAC, and record every use. That way, skipping analysis is a deliberate decision someone owns, not a habit.

What happens during automated rollback, and how do you test it?

When an analysis run fails, the controller aborts the rollout. It sets canary weight to zero so all traffic goes back to the stable ReplicaSet, scales the canary pods down, and marks the Rollout Degraded. The stable pods were never removed, so recovery takes seconds and needs no new scheduling. The Rollout spec still references the bad image, though. It stays aborted until you push a fixed image or run kubectl argo rollouts retry rollout.

Rollback only moves traffic and pods. It does not undo a schema migration, a message already published to a queue, or a cache entry written in a new format. The stable version has to tolerate whatever the canary left behind. Use expand-and-contract migrations, and keep event schemas additive.

To see why a rollout failed, list the runs with kubectl get analysisrun. Then kubectl describe the failed one. It shows each metric's measured values, the evaluated condition, and the phase. kubectl argo rollouts get rollout <name> --watch shows step progress, weights, and ReplicaSet state as they change. The same plugin provides promote, abort, and retry.

Test the failure path before you depend on it. Build an image that returns errors on a fraction of requests or adds latency, deploy it with kubectl argo rollouts set image, and watch the run. If it promotes, your queries or thresholds are too loose.

Once this works, nobody has to watch dashboards after a deploy and revert by hand. The decision is made the same way every time, and the evidence is recorded in the AnalysisRun.

When should you avoid Argo Rollouts, and what are the alternatives?

That record pays off for teams with steady traffic, trustworthy metrics, and several releases a week. Under those conditions a canary step sees enough requests to separate a real regression from noise, and the setup cost is spread across many deploys.

Avoid it, or keep it minimal, when:

  • Traffic is low. A 10% canary that handles a few dozen requests cannot produce a statistically meaningful error rate, so analysis either passes everything or flaps.
  • Risk sits in data or rare paths. A bad schema migration or a broken monthly-billing endpoint will not show up in aggregate latency or error metrics during a 15-minute canary.
  • Failures are slow. A memory leak that takes hours to cause OOM kills outlasts the analysis window. The rollout is already marked healthy by then.
  • The team is small. The CRD, the controller, AnalysisTemplates, and the habit of debugging a paused rollout are ongoing work that someone has to own.
Approach Good fit Main limitation
Deployment rolling update Low-traffic or low-risk services No metric gate, so a human must notice and roll back
Blue-green (in Rollouts) Fast full cutover with instant revert Briefly doubles capacity and gives no partial exposure
Feature flags Per-feature or per-user risk, including rare paths Flag debt and code-level complexity
Other progressive delivery controllers Teams already standardized on a mesh or ingress Same metric-quality dependency

Before adopting it, estimate how many incident hours per quarter a fast metric-gated rollback would have saved. If that number is smaller than the hours spent maintaining analysis queries and the controller, it is not worth adopting yet.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to Argo Rollouts canary, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.