What is a canary deployment for a monolith, and how does it differ from service-level canaries?

A canary deployment for a monolith runs one new, immutable build alongside the current baseline. The new build receives a small, controlled slice of production traffic, and the baseline keeps serving everything else. Both builds come from the same pipeline. The only difference between them is the version being released.

The contrast with service-level canaries matters. In a microservices system, a canary usually exercises one service while its neighbors stay fixed, so a failure points to a narrow change. In a monolith, every release replaces the whole application at once. That means the canary tests the entire build: every module, dependency bump and config change that shipped together. It also means the unit of rollback is the full application version. You cannot back out just the billing change and keep the search fix. You return to the previous build or you don't.

Teams use monolith canaries for two reasons:

  • Limit blast radius. A bad release reaches only a small share of users before someone notices.
  • Catch regressions with real traffic. Some problems never show up in staging. Real request mixes, real data shapes and real load expose them.

The approach fits teams that ship a single deployable often enough that releasing to the full fleet at once has become a real risk.

The rest of this guide is not mainly about the load balancer. Splitting traffic is the easy part. The hard constraints are the database schema, background workers and session state, because both versions share all three while the canary runs.

How does traffic splitting work for a single deployable?

There are three common ways to split traffic, and most teams use them in sequence.

Option Granularity Operational cost Main risk
Weighted upstreams in the existing pool Limited by instance count Low Steps are coarse
Separate canary pool with load balancer weights Any percentage Extra capacity and a second deploy target A lightly loaded pool hides contention bugs
Header or cookie routing for internal users Exact cohort Low Internal traffic is not representative

Instance count matters because each instance is one slice of traffic. As an illustrative estimate, if the monolith runs on eight large instances and you replace one, your smallest step is 12.5%. A dedicated canary pool removes that limit, because the weight sets the share regardless of how many instances serve it. The cost is paying for that extra capacity.

Sticky assignment

Assign users to a version, not individual requests. With per-request random routing, one user bounces between versions. Page markup and static assets can come from different builds. Cache entries get written in one format and read in another. CSRF or session tokens issued by one version may be rejected by the other.

Pin users in one of two ways:

  • Cookie: set it on first assignment.
  • Stable user ID hash: hash the ID into a bucket from 0 to 99 and compare it with the canary percentage. For logged-in users, this stays consistent across devices and cleared cookies.

Order the proxy rules so an explicit override cookie is checked before the weights. This sends internal users to the canary first.

Stickiness has costs:

  • Skewed sample: heavy users with long sessions are overrepresented.
  • Slower draining: pinned sessions keep the canary busy after you set its weight to zero.
  • Rollback: it must expire or ignore the pin.

Both pools share one database, cache and queue.

# http {} context. Two upstream groups share one database, cache and queue.
upstream baseline {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}
upstream canary {
    server 10.0.2.10:8080;
}

# Sticky key: prefer a stable user ID cookie, fall back to client IP.
map $cookie_uid $assign_key {
    ""      $remote_addr;
    default $cookie_uid;
}

# Hash the key into a bucket. The same user always gets the same pool.
# Change the salt string only if you deliberately want to reshuffle users.
split_clients "release-2024-canary:${assign_key}" $weighted_pool {
    5%   canary;
    *    baseline;
}

# Check the explicit override cookie before the weights (internal users first).
map $cookie_release_track $target_pool {
    canary   canary;
    baseline baseline;
    default  $weighted_pool;
}

server {
    listen 80;
    location / {
        proxy_pass
        # Expose the assignment so dashboards and support can see which build served a request.
        add_header X-Release-Track $target_pool always;
    }
}

An nginx config that gives baseline and canary separate upstream pools, sends 5% of users to the canary by hashing a stable user ID (split_clients), and lets a release_track override cookie take precedence over the weights so internal users reach the canary first.

How do you keep the database compatible while two versions run side by side?

Both versions read and write the same database for the entire canary window, and during a rollback. Any schema change the old version cannot tolerate breaks production at the moment of deploy, before your canary metrics notice anything. Every migration must be backward-compatible, so old code still works against the new schema, and forward-compatible, so new code still works against rows the old code writes.

Expand and contract: renaming a column

Renaming customer_name to full_name takes several deploys:

  1. Expand: add full_name as nullable. Neither version uses it yet.
  2. Dual-write: ship code that writes both columns and reads customer_name. Backfill existing rows.
  3. Switch reads: ship code that reads full_name and still writes both.
  4. Stop writing customer_name once no running or rollback-eligible version reads it.
  5. Contract: drop the old column in a separate deploy.

Run migrations as their own pipeline step, before the code that depends on them. If a migration and a code change ship together, a rollback reverts the code but leaves the schema behind.

Shared-state leaks

The database is only one shared surface. Caches and queues are shared too. If the canary changes a serialized object's shape, the stable pool may read a cache entry it cannot parse. Put a version in cache keys and message payloads, and make consumers ignore fields they do not recognize.

The cost is real. Schema changes take more deploys and more calendar time. Someone also has to remember to remove the old column. Track each pending contract step as a ticket so the cleanup actually happens.

-- Each block ships as its own migration, in its own deploy, in this order.
-- Application deploys (dual-write, switch reads, stop writing) happen between them.

-- Migration 1 (expand): additive and nullable, so the running version is unaffected.
-- Both stable and canary can run against this schema.
ALTER TABLE customers ADD COLUMN full_name text;

-- Migration 2 (backfill): run after the dual-write code is fully rolled out,
-- so no new rows are created with full_name left NULL.
-- Re-run until it updates 0 rows. Small batches keep locks short.
UPDATE customers
SET full_name = customer_name
WHERE id IN (
  SELECT id FROM customers
  WHERE full_name IS NULL AND customer_name IS NOT NULL
  LIMIT 5000
);

-- Check before switching reads: this should return 0.
SELECT count(*) FROM customers
WHERE full_name IS NULL AND customer_name IS NOT NULL;

-- Migration 3 (contract): only after no running or rollback-eligible
-- version reads or writes customer_name. Rolling back past this point
-- requires a forward fix, not a code revert.
ALTER TABLE customers DROP COLUMN customer_name;

PostgreSQL migrations for renaming customer_name to full_name with expand and contract, split into separate deploys so that both the stable and canary versions keep working at every step.

What should happen to background workers, schedulers and queue consumers during a canary?

An HTTP router only decides where requests go. Workers, schedulers and queue consumers pull their own work, so traffic splitting does nothing for them. You have to choose explicitly: give them their own canary slice, or pin them to the old version until promotion.

For queue consumers, a slice is usually simple. If you run ten consumer processes, start one or two from the new build. Because consumers compete for messages on the same queue, the new version picks up roughly a proportional share. This only works if message formats are compatible in both directions. Old consumers must handle messages written by new producers, and the reverse, for the same reason the schema must tolerate both versions.

Scheduled jobs need different handling. If every instance runs cron, the job fires once per process, and during a canary that includes both versions. There are two fixes. You can restrict scheduling to a single role, meaning one dedicated scheduler deployment. Or you can have each run acquire a lock before doing any work. In PostgreSQL, pg_try_advisory_lock with a fixed key per job returns immediately: the instance that gets the lock runs the job, and the others skip it. A session-level lock is released automatically when its connection closes, so a crashed holder does not block the job forever.

Holding workers on the old version is simpler, but it has a cost. The new async code then runs for the first time at full rollout, with no canary protecting it. Retry logic, serialization and batch jobs are common sources of incidents. Prefer a small consumer slice when you can.

import os
import psycopg2

# One fixed key per job. Old and new builds must use the same value
# so both versions compete for the same lock during the canary.
NIGHTLY_INVOICES_LOCK_KEY = 4210001


def run_nightly_invoices(conn):
    ...  # the actual job body


def main():
    conn = psycopg2.connect(os.environ["DATABASE_URL"])
    # Autocommit keeps this a plain session-level lock, not tied to a transaction.
    conn.autocommit = True
    try:
        with conn.cursor() as cur:
            # Returns true/false immediately instead of waiting.
            cur.execute("SELECT pg_try_advisory_lock(%s)", (NIGHTLY_INVOICES_LOCK_KEY,))
            (acquired,) = cur.fetchone()
        if not acquired:
            print("nightly_invoices: another instance holds the lock, skipping")
            return
        try:
            run_nightly_invoices(conn)
        finally:
            with conn.cursor() as cur:
                cur.execute("SELECT pg_advisory_unlock(%s)", (NIGHTLY_INVOICES_LOCK_KEY,))
    finally:
        # If the process crashes, the session ends and Postgres drops the lock.
        conn.close()


# Note: the lock prevents overlapping runs, not sequential duplicates. If one
# instance finishes before another's cron tick fires, record the last run
# (e.g. a job_runs row keyed by date) and check it inside the lock.
if __name__ == "__main__":
    main()

A cron entrypoint that uses PostgreSQL's pg_try_advisory_lock with a fixed per-job key so that only one instance, old or new version, runs the scheduled job while the others skip it.

How do you decide whether a canary is healthy enough to promote?

Compare the canary against a concurrent baseline, meaning old-version instances serving the same traffic mix during the same window. Last week's numbers carry different load, different cache state and different upstream behaviour, so a difference may have nothing to do with the release. Track error rate, p50/p99 latency, saturation (CPU, memory, connection pool usage) and two or three business signals such as checkout completions or login success.

Illustrative estimate (all inputs assumed): the monolith serves 200 req/s, and the canary gets 5%, or 10 req/s. A 30-minute soak gives the canary 18,000 requests. At a 0.2% baseline error rate you expect about 36 errors, so a rise to 0.3% (about 54 errors) is visible. A rare endpoint carrying 0.5% of traffic gets only 90 requests. At that baseline you expect fewer than one error there, so even a fivefold regression on that endpoint looks like noise. For rare paths, use longer soaks, synthetic checks or targeted tests.

A staged plan, each stage with a minimum soak and an explicit gate:

  1. Internal users, 1 hour. Gate: no new error classes.
  2. 5%, 1 hour. Gate: error rate and p99 within agreed bounds of the baseline.
  3. 25%, 2 hours, including a peak period. Gate: saturation and business metrics hold.
  4. 100%.
Approach Works well when Fails when
Manual approval Metrics are noisy and the team is still learning what normal looks like Reviewers glance at dashboards and approve by habit
Automated analysis Metrics are stable, low-noise and have tested thresholds Flaky gates block releases until people bypass them, which means promoting blindly

How do you roll back a monolith canary safely, and how should the pipeline be shaped?

Rolling back traffic is the fast part: point the router back at the stable pool and new requests stop reaching the canary within seconds. That only restores the previous state if the canary left nothing behind that the old version cannot handle. Moving traffic does not undo:

  • Schema changes the old code can't read. If the canary's migration dropped or renamed a column, the stable pool breaks the moment it takes all traffic again. This is why expand-only migrations matter during the canary window.
  • Irreversible side effects. Emails already sent, payments captured, and messages published in a new format stay in the world. Consumers on the old version may reject or misread those messages.
  • Poisoned caches. Entries written by the canary in a new shape or with wrong values keep being served to the stable pool until they expire. You may need versioned cache keys or a targeted purge.
  • Corrupted shared data. Rows the canary rewrote incorrectly need a repair script, not a traffic switch.

Feature flags cover what routing cannot. Wrap risky code paths, such as a new billing calculation, behind a flag so you can turn off that one path without redeploying, even after promotion.

Shape the pipeline around one immutable artifact:

  1. Build once and tag the artifact by commit digest.
  2. Deploy to canary by sending that artifact to the canary pool at a small traffic weight.
  3. Analyze canary metrics against baseline metrics for a fixed window.
  4. Promote the same digest to the stable pool, or roll back.

Never rebuild between stages. A rebuild can pull different dependencies, and then the artifact you promote is not the one you tested.

# Generic pipeline pseudocode: one immutable artifact, promoted by digest.
# Not tied to any specific CI system; step names are illustrative.
stages:
  - name: build
    steps:
      - run: build-image --tag app:${COMMIT_SHA}
      # Capture the content digest; every later stage references this, not a tag.
      - export: ARTIFACT=registry.example.internal/app@${IMAGE_DIGEST}

  - name: migrate-expand
    # Expand-only: add columns/tables; never drop or rename during the canary window.
    steps:
      - run: run-migrations --phase expand --artifact ${ARTIFACT}

  - name: deploy-canary
    steps:
      - run: deploy --pool canary --artifact ${ARTIFACT}
      - run: set-traffic-weight --canary 5 --stable 95

  - name: analyze
    steps:
      # Compare canary vs baseline over a fixed window; fail the stage on regression.
      - run: compare-metrics --canary canary --baseline stable --window 30m
          --max-error-rate-delta 0.5% --max-p99-latency-delta 10%
    on_failure:
      - run: set-traffic-weight --canary 0 --stable 100
      - run: set-flag billing_v2 off   # kill switch for the risky path
      - run: purge-cache --key-prefix "v2:"   # drop entries written in the new shape

  - name: promote
    requires: manual-approval
    steps:
      # Same digest that was analyzed; no rebuild between stages.
      - run: deploy --pool stable --artifact ${ARTIFACT}
      - run: set-traffic-weight --canary 0 --stable 100

  # Contract migrations (drops/renames) run in a later release,
  # once no running version still depends on the old schema.

Generic pipeline pseudocode that builds one artifact, references it by digest through canary deployment and analysis, and promotes that same digest; the rollback path resets traffic, turns off a feature flag and purges versioned cache keys.

When should you avoid monolith canaries, and what are the alternatives?

Canaries give false confidence when the canary slice sees too little traffic to separate signal from noise. At 5% of a low-volume app, a rare error path may never run during the whole window. They also cost more than they return in four other cases:

  • You have no spare capacity for a second pool.
  • Your dashboards cannot split metrics by version.
  • Most releases carry schema changes that cannot be made backward compatible.
  • The risky code runs in workers whose side effects, such as sent emails or captured payments, are irreversible before any metric moves.
Approach Risk exposure Speed Cost
Canary Small slice, real traffic Slow, gated Extra pool, metric gates
Blue-green Full fleet at cutover, instant switch back Fast Double capacity during cutover
Rolling deploy Growing share, weak gating Moderate Little extra capacity
Feature-flag-only Scoped per feature; deploy itself unguarded Fast Flag debt, larger test matrix
Dogfooding rings Internal users first Slowest Needs representative internal usage

The development cost of canaries is real:

  • capacity for a second pool
  • expand-contract discipline on every migration
  • longer windows where two versions run side by side
  • gate thresholds that someone must tune and triage

The return is fewer full-fleet incidents, regressions detected while they affect a small fraction of users, and rollbacks that only drain a small pool. The trade pays off when bad releases are frequent or expensive. It rarely pays off when they are neither.

Readiness checklist

  • Enough canary traffic to exercise critical paths within the window
  • Headroom to run both pools
  • Error and latency metrics tagged by version
  • Migrations routinely expand-contract
  • Workers pinned to stable or made idempotent
  • Promotion and rollback automated, not manual

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to canary deployment monolith, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.