What is an Argo Rollouts canary and how does it replace a Deployment?

Progressive delivery means shipping a new version to a small slice of production traffic, checking whether it behaves, and widening exposure only when it does. Argo Rollouts implements this with a custom resource, Rollout, that takes the place of a Deployment. The spec looks familiar: replicas, a selector, and the same pod template. The difference is the strategy block. Instead of a rolling update, a canary strategy lists ordered steps, typically setWeight to shift a percentage of traffic, pause to hold for a fixed duration or until someone promotes manually, and analysis to run metric queries that decide whether the rollout continues.

This suits backend and platform teams that release several times a day to services with steady request traffic. Without automation, each release needs an engineer watching error rates and latency, deciding by eye whether a blip matters, and rolling back by hand. An analysis step runs the same queries against fixed thresholds on every release. It aborts without waiting for a human to notice, so engineers no longer have to watch every deploy.

Migrating without downtime

You have two options. The first is to create a Rollout that references your existing Deployment through workloadRef, which reuses its pod template. The second is to convert the manifest by changing the kind to Rollout and adding a canary strategy. In both cases the Rollout brings up its own pods while the old Deployment keeps serving. Once the Rollout reports healthy, scale the old Deployment to zero, then delete it.

# Rollout that adopts the pod template of an existing Deployment via workloadRef.
# The Deployment keeps serving while the Rollout brings up its own pods.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
  namespace: shop
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: checkout-api        # must match the Deployment's pod labels
  workloadRef:                 # reuse the pod template instead of copying it
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  strategy:
    canary:
      # Without a traffic router, weight is approximated by the
      # ratio of canary pods to stable pods.
      steps:
        - setWeight: 20
        - pause: {duration: 5m}     # let real traffic hit the canary
        - analysis:                 # metric gate: abort if queries fail
            templates:
              - templateName: http-error-rate
        - setWeight: 50
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: http-error-rate
        # After the last step the canary becomes the new stable version.
# Once the Rollout reports Healthy, scale the old Deployment to zero:
#   kubectl -n shop scale deployment/checkout-api --replicas=0
# Keep the Deployment object while workloadRef points at it.

A minimal canary Rollout that reuses an existing Deployment's pod template through workloadRef and gates each traffic increase on an analysis step. It ends with the command to scale the old Deployment to zero once the Rollout is healthy.

How does canary traffic splitting work with and without a traffic router?

Without a traffic router, Argo Rollouts has only one lever: replica counts. Both ReplicaSets sit behind the same Service, and kube-proxy spreads connections roughly evenly across pods. A setWeight: 20 step on a 10-replica Rollout becomes 2 canary pods and 8 stable pods. The controller rounds to whole pods, so a 5-replica service cannot express 10%. Long-lived connections and uneven per-pod load also mean the observed share can drift from the target.

A traffic router separates the share of traffic from the number of pods. You define two Services, a stable Service and a canary Service, each selecting one ReplicaSet through a pod-template-hash label that the controller manages. At each step, the controller rewrites the router's weights, such as ingress controller canary annotations or a mesh route. One canary pod can then receive exactly 5% of requests.

            clients
               │
        ┌──────▼──────┐
        │   router    │  weights set by controller
        │ (ingress /  │
        │    mesh)    │
        └──┬───────┬──┘
       95% │       │ 5%
   ┌───────▼──┐ ┌──▼───────┐
   │ stable   │ │ canary   │
   │ Service  │ │ Service  │
   └───────┬──┘ └──┬───────┘
   ┌───────▼──┐ ┌──▼───────┐
   │ stable RS│ │ canary RS│
   └──────────┘ └──────────┘
Aspect Replica-based weighting Router-based splitting
Weight precision Limited by replica count Arbitrary percentages
Minimum canary share 1 / total replicas As low as the router allows
Extra components None Ingress controller or mesh
Configuration Single Service Stable and canary Services plus router reference
Request-level routing (headers) Not possible Possible where the router supports it

The trade-off is that the router becomes part of the release path. Its misconfiguration or downtime affects rollouts, and the Rollout spec must match the router's resources exactly.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 4               # pod count no longer limits the canary share
  selector:
    matchLabels:
      app: checkout
  template:
    metadata:
      labels:
        app: checkout
    spec:
      containers:
        - name: checkout
          image: registry.example.com/checkout:1.4.0
          ports:
            - containerPort: 8080
  strategy:
    canary:
      # Two existing Services; the controller injects the
      # rollouts-pod-template-hash selector into each one
      stableService: checkout-stable
      canaryService: checkout-canary
      trafficRouting:
        nginx:
          # Ingress that routes to checkout-stable; the controller
          # creates a canary Ingress and manages its weight annotation
          stableIngress: checkout
      steps:
        - setWeight: 5      # 5% of requests, not 1 of 4 pods (25%)
        - pause: {duration: 10m}
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {}         # wait for manual promotion

A Rollout using the NGINX ingress traffic router with separate stable and canary Services, so a 4-replica deployment can still send exactly 5% of requests to the canary.

How do you write an AnalysisTemplate that gates promotion?

An AnalysisTemplate is a reusable list of metrics. Each metric is queried on a schedule and judged by expressions, and the Rollout references the template by name. Start with spec.args, which declares the inputs. The Rollout supplies values for them, and the most useful one is the canary's pod template hash, set with valueFrom.podTemplateHashValue: Latest. Interpolate it into the query as {{args.canary-hash}} and filter on a label that carries the hash, such as rollouts_pod_template_hash. That way the query measures only canary pods. A service-wide error rate dilutes a bad canary with healthy stable traffic.

Each entry under metrics has these fields:

  • provider: the backend and query. For example, prometheus.address and prometheus.query, with the result exposed as result.
  • successCondition / failureCondition: expressions such as result[0] < 0.01. If you define both and neither matches, the measurement is Inconclusive.
  • interval and count: how often to measure and how many times. Together they set how long the gate observes the canary.
  • failureLimit: how many failed measurements to tolerate before the analysis fails.
  • inconclusiveLimit: how many inconclusive results to tolerate. Exceeding it pauses the rollout for a human to decide, rather than promoting or aborting.

Handle missing data deliberately. A Prometheus query over a canary that has served no requests returns an empty vector. Appending or vector(0) turns that silence into a passing value. Instead, write conditions so that "no data" falls into the inconclusive band. Query errors, such as an unreachable backend, are counted separately, and consecutiveErrorLimit decides how many are tolerated before the metric errors out. Set it high enough to ride out a brief blip, but keep it finite so a dead metrics backend stops the release.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-error-rate
spec:
  args:
    - name: service-name
    # Supplied by the Rollout via valueFrom.podTemplateHashValue: Latest
    - name: canary-hash
  metrics:
    - name: error-rate
      interval: 1m              # measure once a minute
      count: 10                 # 10 measurements, about 10 minutes of observation
      failureLimit: 1           # tolerate one bad reading, abort on the second
      inconclusiveLimit: 2      # more than 2 unclear readings pauses for a human
      consecutiveErrorLimit: 5  # ride out a short metrics-backend outage
      # Empty vector (no traffic yet) matches neither condition and is
      # Inconclusive. NaN (0/0) fails both comparisons and is Inconclusive too.
      # Readings between 1% and 5% are also Inconclusive.
      successCondition: len(result) > 0 && result[0] < 0.01
      failureCondition: len(result) > 0 && result[0] >= 0.05
      provider:
        prometheus:
          address:
          # Filter on the canary hash so stable traffic cannot dilute the signal.
          # No 'or vector(0)': silence must not read as a pass.
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",
              rollouts_pod_template_hash="{{args.canary-hash}}",code=~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}",
              rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))

An AnalysisTemplate that scopes a Prometheus error-rate query to canary pods through a pod-template-hash argument. Its conditions are written so that an empty result or NaN counts as inconclusive instead of passing, and consecutiveErrorLimit absorbs brief backend outages.

Which metrics should gate a canary deployment?

Gate on what users experience, not on what the pods consume. CPU and memory tell you the new version is busier, but a busier canary can be perfectly healthy, and a broken one that returns fast 500s can look idle. Error rate and latency percentiles map directly to user harm. Scope them to canary pods, for example by filtering on the pod-template-hash label or a canary-specific service. If you skip that, stable traffic dilutes the signal and a canary at 5% weight barely moves the aggregate.

Absolute thresholds such as "p99 under 400 ms" work when the baseline is steady. When latency swings with time of day or downstream load, a fixed number either fires on normal noise or sits so high that it means nothing. Comparing the canary to stable over the same window cancels shared noise. A rule like "canary error ratio no more than stable plus 0.5 percentage points" asks whether the new version is worse, which is the real question.

Tuning cuts both ways. If the gate is too tight, rollouts abort on noise and engineers learn to override them. If it is too loose, real regressions get promoted. Before you set any limit, run the queries against stable for a week or two and record the actual spread.

Metric What it catches Common pitfalls
5xx error ratio Crashes, bad config, broken dependencies Undefined when there are zero requests. Ignores 4xx caused by contract changes
p95/p99 latency Slow queries, lock contention, cold caches Noisy at low volume. Histogram bucket boundaries distort the result
Business success rate (checkouts, logins) Logic bugs that still return 200 Sparse data. Lags behind the release
CPU/memory Leaks, runaway loops Weak link to user impact. Changes legitimately between versions
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-ratio-vs-stable
spec:
  args:
    # Filled from the Rollout so each query sees only one ReplicaSet's pods.
    - name: canary-hash
      valueFrom:
        podTemplateHashValue: Latest
    - name: stable-hash
      valueFrom:
        podTemplateHashValue: Stable
  metrics:
    - name: canary-5xx-delta
      interval: 1m
      count: 10
      # Tolerate one noisy sample before aborting.
      failureLimit: 1
      # Gate on the gap between canary and stable, not on an absolute ratio:
      # fail if canary is more than 0.5 percentage points worse.
      # Set 0.005 only after observing stable's real spread for a week or two.
      successCondition: result[0] <= 0.005
      provider:
        prometheus:
          address:
          # Assumes scrape relabeling copies the pod-template-hash pod label
          # onto request metrics as pod_template_hash.
          # 'or vector(0)' keeps the query defined when there are no 5xx;
          # clamp_min avoids dividing by zero when there is no traffic.
          # Zero canary traffic then passes vacuously, so check volume separately.
          query: |
            (
              (sum(rate(http_requests_total{app="checkout",pod_template_hash="{{args.canary-hash}}",code=~"5.."}[5m])) or vector(0))
              / clamp_min(sum(rate(http_requests_total{app="checkout",pod_template_hash="{{args.canary-hash}}"}[5m])), 1e-9)
            )
            -
            (
              (sum(rate(http_requests_total{app="checkout",pod_template_hash="{{args.stable-hash}}",code=~"5.."}[5m])) or vector(0))
              / clamp_min(sum(rate(http_requests_total{app="checkout",pod_template_hash="{{args.stable-hash}}"}[5m])), 1e-9)
            )

This Argo Rollouts AnalysisTemplate scopes the 5xx error ratio to the canary and stable ReplicaSets by pod-template-hash, then fails the canary only if its error ratio exceeds stable's by more than 0.5 percentage points over the same window.

When should you use background analysis versus inline analysis steps?

Background analysis is declared once at the strategy level and runs alongside the steps for the whole rollout. It aborts the rollout as soon as a measurement crosses its failure limit. You can delay its start until a given step index, so it doesn't evaluate a canary that receives no traffic yet. Inline analysis is itself a step. The rollout stops at that step until the AnalysisRun finishes, then advances or aborts based on the result.

Background analysis Inline analysis step
Scope Entire rollout, from its starting step onward One point in the step list
Blocks progression No, it only aborts on failure Yes, until the run completes
Best for Fast, continuous signals such as error rate and p99 latency Slower or costlier checks before a risky jump
Main risk Too few samples early in the rollout Adds wall-clock time to every release

A practical combination is to run a background error-rate and latency check from the first traffic step onward. Then add an inline gate before the biggest exposure change, such as going from 10% to 50%. That gate is where a business success-rate query fits, because it needs several minutes of data to mean anything.

Every inline gate and pause adds to lead time. A six-step rollout with ten-minute gates means at least an hour per release, including for a one-line hotfix. Document a fast-track path in advance. kubectl argo rollouts promote --full skips the remaining steps and analysis. Agree on who may run it and under what conditions, and record each use. That way, urgent fixes don't lead engineers to bypass the rollout entirely with ad hoc manifest edits.

What happens when a canary gate fails, and how do you operate the rollout?

When an analysis metric crosses its failure limit, the controller aborts the Rollout. It shifts traffic back to the stable ReplicaSet, scales down the canary, and sets the Rollout's phase to Degraded. The spec still names the new image, though. The Rollout stays aborted until someone retries it or changes the spec, so a failed release does not silently try again.

To find out why the gate failed, start with kubectl argo rollouts get rollout <name>, which shows the failed AnalysisRun. Then run kubectl describe analysisrun <run-name>. It lists each metric's measurements, the value returned, and the phase. The phase tells you what kind of problem you have:

  • Failed: the condition evaluated false. The canary looks bad.
  • Error: the query itself broke, for example because the metrics provider was unreachable or the query was malformed. This is a problem with the gate, not the release.
  • Inconclusive: the result fell between your success and failure conditions. This pauses the rollout for a human decision instead of aborting it.

For daily operation, a few plugin commands cover most cases:

  • get rollout <name> --watch follows a release step by step.
  • promote resumes a paused step.
  • abort stops a release manually.
  • retry rollout restarts an aborted release.

promote, abort and retry act on the Rollout's status, so a GitOps tool syncing the manifest won't see them as drift. Commands that change the spec, such as set image, are different. A sync with self-heal will revert them, or the application will show as out of sync.

Treat Git as the place where you decide what runs. After an abort, revert or fix the image tag in Git rather than retrying by hand. Use retry only when the gate itself errored and the release is believed to be sound.

How many canary requests does a gate actually evaluate on low-traffic services?

Every gate decision rests on however many requests reached the canary during the measurement window. On a quiet service that number can be very small.

Illustrative estimate. Assume a service averaging 2 requests per second, a canary step at 10% weight, a 5-minute pause, and an analysis that measures every minute over a 1-minute query window. The canary receives about 0.2 requests per second. Each measurement therefore covers about 12 requests, and the whole step covers about 60.

Total RPS (assumed) Canary weight Canary requests per 1-min measurement Canary requests per 5-min step
2 5% ~6 ~30
2 10% ~12 ~60
2 25% ~30 ~150
200 10% ~1,200 ~6,000

At 12 requests, a single failure produces an 8.3% error rate. That breaches a 5% threshold, so one unlucky timeout aborts a healthy release. The opposite failure is just as likely. Zero errors in 60 requests is consistent with a true error rate of several percent, because the rough 95% upper bound for zero failures in n trials is about 3/n, which here is 5%. The gate passes, but it has not shown that the canary is healthy.

You can fix this in three ways:

  • Require a minimum request count in the query. Below that count, return a result that matches neither the success nor the failure condition, so the measurement is inconclusive instead of a false pass or fail.
  • Lengthen the pauses and query windows. Size them to collect enough requests for the threshold you care about.
  • Add synthetic traffic. Send a steady stream of realistic requests to the canary so the sample size no longer depends on organic load.

When should you avoid metric-gated canaries, and what are the alternatives?

A metric-gated canary only catches regressions that show up in the metrics you query, during the window you watch, on the traffic the canary actually receives. That leaves gaps:

  • Data migrations and schema changes. A migration runs once against shared storage, so there is no 10% version to roll back. If it corrupts data, aborting the rollout leaves the damage in place.
  • Rarely used paths. A monthly billing job or an admin endpoint may get no requests during analysis, so the gate passes without testing that code.
  • Version skew. Stable and canary run side by side. Both must read and write the same database rows, queue messages, caches and API contracts. That means backward- and forward-compatible changes, often split across several releases.

The operational cost is real. You run another controller and its CRDs, and you upgrade both. Some dashboards, policy checks and scripts expect a Deployment and may not recognise a Rollout. The gate is also only as reliable as your metrics backend. If queries are slow or return no data, releases stall or fail. Writing and tuning queries takes engineering time. That cost pays off when you release often and someone would otherwise watch dashboards and roll back by hand. It rarely pays off for a service you ship once a quarter.

Approach Protects against Main cost
Rolling update Pods that fail readiness probes Little protection against bad behaviour that passes probes
Blue-green Bad releases, with fast full cutover and rollback Double capacity during switch; all users move at once
Feature flags Logic bugs, including on rare paths, toggled per user Flag sprawl, stale-flag cleanup, flag service dependency
Metric-gated canary Regressions visible in aggregate metrics Controller overhead, query tuning, metrics dependency

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to argo rollouts canary, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.