What is a canary release for a monolith, and how is it different from a microservice canary?

A canary release for a monolith means running a second version of the whole application next to the current one and sending it a small slice of production traffic. A load balancer, ingress or router decides which requests reach the canary. Everything else stays on the stable version until you choose to promote or abort.

A microservice canary usually ships one service and one narrow change. A monolith deploy is different. It typically bundles weeks of merged work: several features, dependency upgrades, config changes and migrations in one artifact. That has two effects:

  • The blast radius is wider. Any code path in the application might be the one that broke.
  • The signal is noisier. If the canary's error rate rises, the dashboard does not tell you which of twenty changes caused it.

The core point of this guide is that traffic splitting is the easy part. The hard part is shared state. Both versions read and write the same database schema, consume from the same queues, populate the same caches and decode the same session data. A monolith canary is only safe if the old and new code can both operate on that state at the same time.

Teams accept this complexity because a canary finds problems that staging misses:

  • Build regressions, such as a missing asset or a bad dependency.
  • Runtime regressions, such as memory growth, slow queries or connection pool exhaustion.
  • Behavior regressions that only real users and real data trigger.

The approach fits backend and platform teams that meet three conditions:

  • They run a single deployable.
  • They have enough traffic for a small slice to produce meaningful metrics.
  • They have an on-call owner who watches the rollout and can stop it.

How does traffic splitting work for a single deployable?

Every request takes the same path. A load balancer or reverse proxy receives it and forwards it to one of two instance pools: baseline (current build) or canary (new build). Both pools read and write the same database, queues and cache. The split happens only at the routing layer, so anything downstream of it is shared and has to tolerate both versions.

Method Granularity Stickiness Main weakness
Instance-count ratio behind a load balancer Coarse, steps of 1/N None by default 1 canary out of 4 instances is already 25%
Weighted routing at a proxy or ingress Fine (1%, 5%) Only if you add hashing Needs proxy config you can change quickly
Header- or cookie-based routing Exact, per user Explicit Covers only users you opt in

Instance-ratio splitting is the easiest to set up. On a small fleet it fails because you can't start below one instance's share of traffic. Weighted routing separates traffic share from instance count, so one canary instance can take 2% of requests. A header override on top lets internal testers reach the canary before any customer does.

Pin each user to one version. If a user bounces between builds, they see UI or validation behaviour change from one click to the next, and their errors get counted against both versions in your metrics. Routing on a hash of the user or session ID keeps each cohort stable. The trade-off is that a fixed hash bucket may over-represent certain tenants, regions or endpoints. Check which traffic the canary actually received before you trust its metrics.

The example config shows weighted upstreams with a header override for testers.

# http {} context. Baseline and canary are separate pools of the same app.
upstream app_baseline {
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

upstream app_canary {
    server 10.0.2.10:8080;   # one canary instance, independent of its traffic share
}

# Stickiness key: session cookie if present, else client IP, so a user
# does not bounce between builds from one request to the next.
map $cookie_session_id $sticky_key {
    ""      $remote_addr;
    default $cookie_session_id;
}

# Hash the key into buckets: about 2% of keys go to the canary.
split_clients "${sticky_key}" $weighted_pool {
    2%   app_canary;
    *    app_baseline;
}

# Internal testers send "X-Canary: 1" to reach the canary first.
# Strip or reject this header from untrusted clients at the edge.
map $http_x_canary $app_pool {
    "1"     app_canary;
    default $weighted_pool;
}

server {
    listen 80;

    location / {
        proxy_pass    # resolves to one of the upstream groups above
        proxy_set_header Host $host;
        add_header X-Served-Pool $app_pool always;   # shows which pool served the request
    }
}

An nginx config that sends a sticky 2% of sessions to a single canary instance using split_clients on a session-ID hash, with an X-Canary header override so internal testers reach the canary before customers.

How do you make old and new versions coexist on shared state?

During a canary, two versions of the monolith run against the same database, queues and caches. Every shared structure must work for both at once, so any change to it has to ship across several releases.

Expand and contract

For the database, use expand-and-contract (also called parallel change).

  1. Expand: add the new column or table, nullable or with a default, so old code's inserts still succeed.
  2. Migrate: have new code write to both shapes or tolerate either. Backfill existing rows in batches, then switch reads to the new structure.
  3. Contract: drop the old column only in a later release, once no running version reads it.

The order matters because rollback means old code runs again. If the release that stops using a column also drops it, aborting the canary sends traffic back to a version that queries a column that no longer exists. Destructive changes are only safe once every version you might roll back to has stopped depending on the old structure.

Beyond the schema

The same rule applies to message payloads, cache entries and serialized sessions:

  • Backward compatible: new code must read old formats.
  • Forward compatible: old code must tolerate new formats.

In practice, readers should ignore unknown fields and supply defaults for missing ones. Put a version in cache keys so the two releases don't overwrite each other's entries with shapes the other can't parse.

The cost

Every release has to be compatible in both directions. A column rename becomes three deployments, and contract changes need planning across several release cycles. That slows schema work, and it's the price of being able to abort a canary safely.

-- Rename users.fullname -> users.display_name across three releases.
-- Every step must be safe while version N and N+1 run side by side.

-- RELEASE 1 (expand): additive only. Old code ignores the new column,
-- and its INSERTs still succeed because the column is nullable.
ALTER TABLE users ADD COLUMN display_name TEXT NULL;
-- Release 1 application code writes BOTH fullname and display_name,
-- and still reads fullname.

-- RELEASE 2 (migrate): backfill in small batches to avoid long locks.
-- Repeat until it updates 0 rows.
UPDATE users
SET display_name = fullname
WHERE id IN (
  SELECT id FROM users
  WHERE display_name IS NULL AND fullname IS NOT NULL
  LIMIT 1000
);
-- Release 2 code reads display_name, falls back to fullname if NULL,
-- and keeps writing both. Rolling back to release 1 is still safe
-- because fullname is still populated.

-- RELEASE 3 (contract): ship only after no running or rollback-target
-- version reads or writes fullname. Release 3 code must have stopped
-- writing fullname before this migration runs.
ALTER TABLE users DROP COLUMN fullname;

An illustrative PostgreSQL-style sequence that renames a column in three releases: add the new column, backfill it in batches while code writes both columns, and drop the old column only after no version you might roll back to still uses it.

How should background jobs, queue consumers and scheduled tasks behave during a canary?

In a monolith, the scheduler and queue consumers usually start inside the same process that serves HTTP. Every canary instance therefore picks up real background work by default, and the HTTP traffic split you configured says nothing about how much of that work it handles.

There are three common choices:

  • HTTP only. Canary instances serve requests but run no jobs. This is the easiest to reason about, but background code gets no canary exposure.
  • Proportional consumption. The canary joins the consumer group and takes roughly its share of messages. Job code gets real exposure, but the canary's share comes from its consumer count, not from your traffic weight.
  • Named subset. The canary runs only the specific jobs whose code changed.

An environment-driven role flag handles all three. For example, set APP_ROLE=web on canary instances and have startup code register the scheduler and consumers only when the role is worker or all. Add an allowlist variable such as CANARY_JOBS=invoice_sync to opt specific jobs back in.

Whatever you choose, assume both versions will see the same work. Use idempotency keys so a retried or duplicated job has no extra effect. Protect each schedule with a leader lock or single-runner election so it fires once, not once per version. Put a version field in job payloads so the old code can reject or skip messages it can't interpret.

A clean canary is weak evidence for rare paths. Month-end billing, admin tools and particular tenants may never run during the canary window, so a quiet dashboard doesn't prove those paths are safe.

import os

# Role is set per deployment, e.g. APP_ROLE=web on canary instances.
# web    -> serve HTTP, run no background work (unless allowlisted)
# worker -> run scheduler and queue consumers, no HTTP
# all    -> default for stable instances: everything in one process
ROLE = os.environ.get("APP_ROLE", "all")

# Opt specific jobs back in on the canary, e.g. CANARY_JOBS=invoice_sync
CANARY_JOBS = {
    name.strip()
    for name in os.environ.get("CANARY_JOBS", "").split(",")
    if name.strip()
}

ALL_JOBS = ["invoice_sync", "nightly_cleanup", "month_end_billing"]


def enabled_jobs():
    if ROLE in ("worker", "all"):
        return ALL_JOBS
    # Web-only instances run nothing except explicitly allowlisted jobs.
    return [job for job in ALL_JOBS if job in CANARY_JOBS]


def start(http_server, scheduler):
    # http_server and scheduler are the app's own objects; only the gating matters here.
    for job in enabled_jobs():
        # Each job should still take a leader lock when it fires,
        # so a schedule runs once across old and new versions.
        scheduler.register(job)
    if enabled_jobs():
        scheduler.start()
    if ROLE in ("web", "all"):
        http_server.serve()

Startup code reads APP_ROLE and an optional CANARY_JOBS allowlist so canary instances serve HTTP with schedulers disabled, except for specific jobs you opt back in.

How do you decide whether to promote, hold or abort a canary?

Compare the canary against the baseline cohort over the same window, not against fixed thresholds or yesterday's numbers. Both cohorts share the same traffic mix, time of day, downstream incidents and cache state, so any gap is more likely caused by the new code. Label every signal by version: error rate, p50/p95/p99 latency, saturation (CPU, memory, connection pool usage, queue lag) and a few business metrics such as logins or checkouts.

Write the schedule and criteria down before deploying:

Step Traffic Min bake Min canary requests Abort if
1 1% 30 min 2,000 Errors > 2× baseline
2 5% 60 min 5,000 p99 > 1.3× baseline
3 25% 60 min 20,000 Checkouts < 0.9× baseline
4 100% n/a n/a n/a

Hold if a step's volume or bake time isn't met, or if a signal is ambiguous.

Illustrative estimate. Assume 50 req/s, a 5% canary (2.5 req/s), a 0.5% baseline error rate, and a goal of detecting a rise to 1.0%. A standard two-proportion sample-size calculation at roughly 95% confidence and 80% power needs about 4,700 canary requests, which takes about 30 minutes. At 2 req/s total, the same 5% step needs about 13 hours. Low-traffic monoliths need larger percentages or longer bakes.

Automated analysis can fail in both directions. It can promote a release whose failure mode no dashboard measures, and it can abort good releases on noisy p99s. Tune thresholds against past deploys and keep a human override.

What does a safe rollback strategy look like for a monolith canary?

Pulling traffic off canary instances takes seconds: change the router weight or drain the new pool, and requests stop reaching the new code. That only helps if the old version can still read everything the new version wrote while it was live. A new enum value, a renamed JSON field or a row in an unfamiliar state can crash old read paths and job processors after the "rollback" is complete.

Some side effects cannot be routed away at all. Data writes persist. Sent emails stay sent. Events already published to a broker will be consumed. Calls to payment providers, partner APIs or webhooks have already happened. A rollback plan has to say how each of these is tolerated, compensated or prevented. "Shift traffic back" does not cover them.

Test the rollback path, not just the rollout

Make rollback a repeatable pipeline step in staging. Deploy the new version, generate realistic writes through its endpoints and jobs, then roll back to the old version. Run the old read paths and job processors against that data. If anything fails to parse or process, the release is not safe to canary, even if it passes every forward test.

Where feature flags fit

The canary validates the build and runtime: dependencies, configuration, memory, startup. Flags handle risky behavior changes, so you can switch off one bad feature without rolling back the whole release. A failing canary tells you the release is bad, not which change caused it. Flags and smaller releases narrow that search, so bisecting takes minutes instead of an afternoon.

#!/usr/bin/env bash
# Rollback rehearsal: run in staging before any canary is allowed.
# The deploy/seed/check scripts are placeholders for your own tooling.
set -euo pipefail

OLD_VERSION="${OLD_VERSION:?set to the currently released build, e.g. 2024.06.1}"
NEW_VERSION="${NEW_VERSION:?set to the candidate build, e.g. 2024.06.2}"
STAGING_URL=""

# 1. Start from the version production runs today.
./scripts/deploy.sh staging "$OLD_VERSION"

# 2. Roll forward to the candidate, including its migrations.
./scripts/deploy.sh staging "$NEW_VERSION"

# 3. Generate realistic writes through the new code: API calls and
#    background jobs, so new enum values, fields and states land in the DB
#    and on the queue.
./scripts/seed_traffic.sh "$STAGING_URL" --scenario full
./scripts/run_jobs.sh --until-queue-empty

# 4. Roll back the application only. Migrations stay applied, as they
#    would in a real rollback.
./scripts/deploy.sh staging "$OLD_VERSION" --skip-migrations

# 5. Exercise the old read paths and job processors against data the
#    new version wrote. Any parse or processing error fails the pipeline.
./scripts/smoke_read_paths.sh "$STAGING_URL"
./scripts/run_jobs.sh --until-queue-empty --fail-on-error

# 6. Check logs from the old version for deserialization failures that
#    did not surface as HTTP errors.
if ./scripts/fetch_logs.sh staging --since 10m | grep -Ei 'unknown enum|deserializ|unexpected state'; then
  echo "Rollback to $OLD_VERSION is unsafe: old code cannot read new data" >&2
  exit 1
fi

echo "Rollback rehearsal passed for $NEW_VERSION -> $OLD_VERSION"

A pipeline step that deploys the candidate in staging, generates writes, rolls back to the old version without reverting migrations, and fails the build if old read paths or job processors cannot handle the new data.

When should you avoid a monolith canary, and what are the alternatives?

A canary pays off when three things hold. The canary cohort must see enough requests to produce a statistically meaningful error and latency signal within a bake time you can tolerate. The team must be able to keep every release backward- and forward-compatible with shared state. And an incident must be expensive enough to justify the extra machinery.

Skip it when those conditions fail:

  • Low traffic. If a 5% cohort sees a few dozen requests an hour, the canary turns into ceremony: you wait, see nothing, and promote anyway.
  • Small teams with good blue-green. A team that already switches environments reliably and can roll back in under a minute gains little from a canary.
  • Irreversible side effects. If a release mostly sends emails, charges cards or runs one-way migrations, the damage happens on the first canary request.
Approach Blast radius Rollback speed Infra cost Shared-state requirement
Blue-green All users at switch Fast (flip back) Double capacity briefly Both versions compatible during switch
Rolling deploy Grows per batch Slow (redeploy) Low Mixed versions throughout
Feature flags / dark launch Per feature, per cohort Instant toggle Low infra, high code discipline Flag-off path must stay valid
Shadow traffic None for reads N/A Extra capacity Writes must be suppressed or isolated
Internal / tenant-by-tenant Chosen tenants Fast if tenant-routed Moderate Tenant isolation in data

The trade-off is predictable. A canary costs extra capacity, routing configuration and per-version metric labels. In return, a regression caught in a small cohort replaces a fleet-wide rollback, an incident response and the debugging that follows.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to canary release monolith, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.