What is a canary deployment for a monolith, and how does it differ from a microservice canary?

A canary deployment for a monolith runs two versions of the same application side by side, both reading and writing the same database, caches, queues and session store. A small share of production traffic, often a few percent, goes to the new version while the rest stays on the current one. If the new version misbehaves, only that slice of users is affected, and you can pull it back before the problem reaches everyone. The goal is to limit the blast radius of a deploy, not to test the code. Testing happens earlier. The canary catches what only real traffic, real data and real load reveal.

The mechanics look like a microservice canary, but the signal is different. When you canary one microservice, the change is confined to that service's API, so a single error rate or latency figure tells you a lot. A monolith deploy changes every code path at once: checkout, search, admin pages and background jobs all ship together. A regression in one low-traffic route can disappear inside a healthy global average. So you have to compare the canary and the baseline per endpoint or per route group, not only in aggregate.

This article's thesis is that a monolith canary is mainly a data-compatibility and routing-stickiness problem. It is not a load-balancer feature. Splitting requests by weight is easy. The hard parts are keeping each user on one version and making sure both versions can safely share state.

It is worth the effort for teams that deploy frequently, have enough traffic to produce a statistically useful signal, and pay a high price for incidents. Within progressive delivery, the canary is the step where exposure grows gradually and each increase depends on evidence from the step before.

                          +---------------------+
                          |   Users / clients   |
                          +----------+----------+
                                     |
                                     v
     +-------------------------------+-------------------------------+
     |  Router: weighted split + stickiness (user pinned to version) |
     +---+-------------------------------------------------------+---+
         |                                                       |
         v                                                       v
+------------------+     +-----------------------+     +------------------+
| Baseline (v1)    |---->| Compare per route:    |<----| Canary (v2)      |
| ~95% of traffic  |     | error rate, latency   |     | ~5% of traffic   |
| every code path: |     | per endpoint/group    |     | every code path: |
| checkout, search,|     | not just global avg   |     | checkout, search,|
| admin, jobs      |     | -> promote/roll back  |     | admin, jobs      |
+--------+---------+     +-----------------------+     +---------+--------+
         |                                                       |
         v                                                       v
     +---+-------------------------------------------------------+---+
     |  Shared state: database, caches, queues, session store        |
     |  Both versions read and write it, so schemas and data formats |
     |  must stay compatible with v1 and v2 at the same time         |
     +---------------------------------------------------------------+

A monolith canary: a sticky router sends a small share of users to v2, every code path changes at once so metrics are compared per route, and both versions share the same state.

How does traffic splitting work, and how do you keep users pinned to one version?

Traffic splitting turns that evidence loop into routing decisions. There are three common mechanisms, and most teams use them in sequence.

Mechanism How it routes Best used for
Header or cookie match An explicit flag sends the request to the canary Internal staff and test accounts before any real users
Weighted upstreams The load balancer or reverse proxy assigns relative weights to the old and new pools Exposing a fixed share of production traffic
Percentage ramp The weights change in steps, such as 1%, 5%, 25%, then 100% Increasing exposure as metrics stay healthy

Plain weighting decides per request, and a monolith cannot tolerate that. A checkout that renders the cart on v2, posts payment to v1, and confirms on v2 can fail on form fields, session serialization, or CSRF tokens that one version produces and the other rejects. OAuth is just as fragile: the state value created before the redirect has to be validated by a version that understands it when the callback arrives.

Pinning solves this by making the routing decision once per user. One option is to hash a stable key, such as a user ID or a session cookie, onto the weighted pools. The other is to set a sticky cookie on the first response and route on it afterwards. Hashing on user ID survives cookie loss and works across devices. Note that changing the weights reshuffles some users unless the proxy uses consistent hashing.

Stickiness has a cost. The sample stops being random. If the hash key is a tenant ID, a single large tenant can fill the whole 5% bucket. Anonymous traffic without a stable key may never reach the canary. A regression that only affects one region, plan tier, or locale can stay hidden until the ramp reaches 100%. To catch these problems, break canary metrics down by segment rather than reading only the aggregate.

# http {} context: pin each user to one pool, with an override for staff.
upstream app_stable { server 10.0.1.10:8080; server 10.0.1.11:8080; }
upstream app_canary { server 10.0.2.10:8080; }

# Pinning key: logged-in user ID from a cookie, else client IP.
# Without a fallback, every anonymous request would hash the same
# empty string and land in a single bucket.
map $cookie_user_id $pin_key {
    ""      $remote_addr;
    default $cookie_user_id;
}

# Hash the key into percentage buckets. The same key always gets the
# same pool. Raising 5% to 25% widens the canary range, so users
# already on the canary stay there.
split_clients "${pin_key}" $weighted_pool {
    5%  app_canary;
    *   app_stable;
}

# Internal users opt in explicitly before any real traffic is exposed.
map $cookie_canary $target_pool {
    "1"     app_canary;
    default $weighted_pool;
}

server {
    listen 80;
    server_name shop.example.com;

    location / {
        # Tag responses so logs and dashboards can split metrics by version.
        add_header X-Served-By $target_pool always;
        proxy_set_header Host $host;
        proxy_pass
    }
}

An nginx config that uses a map for a cookie-based staff override and split_clients to hash a stable user key into a 5% canary bucket, so a user's checkout or OAuth flow stays on one version for every request.

How do you run two versions against one database and shared state?

During a canary, both versions read and write the same rows, sessions, caches and queues. Every change to shared state must therefore work in both directions: the old version has to tolerate whatever the new version writes, and the new version has to tolerate data the old version writes.

Expand and contract the schema

Split each schema change into an expand step and a contract step. In the expand step, add only what the old code can ignore, such as nullable columns, new tables, or indexes. Deploy that migration before the canary starts. Then run the old version's test suite against the migrated schema. This test is the actual proof that rollback is safe.

Keep drops, renames and NOT NULL constraints for the contract step, which runs only after full promotion. If you drop a column mid-canary, rolling back breaks.

Shared state beyond the database

  • Sessions: use a serialization format both versions can read, and ignore unknown fields.
  • Caches: prefix keys with a schema version (for example, v2:user:42). This keeps each version from reading entries whose shape it doesn't understand.
  • Jobs and queues: either version may consume a message the other one produced. Readers should default missing fields and ignore extra ones.
  • Scheduled tasks: run them on exactly one version, enforced with a flag or a lock, so they don't execute twice.

The cost

This approach means extra deploys and temporary code paths. The contract step is often forgotten, and that leaves schema debt behind. Compatibility bugs also tend to appear only while both versions are running, which is exactly the window you are trying to keep safe.

import json

# Expand step: run before the canary starts. The old version never selects
# this column, so it keeps working. Rollback stays safe because nothing was
# dropped. NOT NULL and the drop of any legacy column wait for the contract step.
EXPAND_SQL = """
ALTER TABLE orders ADD COLUMN currency VARCHAR(3) NULL;
"""

# Bump this whenever the shape of a cached value changes. The old version
# reads v1:* and the new version reads v2:*, so neither one deserializes
# an entry it doesn't understand.
CACHE_SCHEMA_VERSION = "v2"

def cache_key(entity: str, entity_id: int) -> str:
    return f"{CACHE_SCHEMA_VERSION}:{entity}:{entity_id}"  # e.g. v2:user:42

# Fields this version understands. Either version may consume a job that
# the other version enqueued during the canary.
KNOWN_FIELDS = {"order_id", "amount_cents", "currency", "notify"}

def read_order_job(raw: bytes) -> dict:
    payload = json.loads(raw)
    # Drop fields added by a newer producer instead of raising.
    job = {k: v for k, v in payload.items() if k in KNOWN_FIELDS}
    # Default fields that an older producer doesn't send yet, or that are
    # NULL for rows written before the expand step.
    if job.get("currency") is None:
        job["currency"] = "USD"
    job.setdefault("notify", True)
    # These fields are required in every version, so fail loudly if missing.
    missing = {"order_id", "amount_cents"} - job.keys()
    if missing:
        raise ValueError(f"order job missing required fields: {sorted(missing)}")
    return job

An expand-only migration, a version-prefixed cache key and a job payload reader that ignores unknown fields and defaults missing ones, so the old and new versions can share the same database, cache and queue during a canary.

Which canary metrics should you choose before deploying?

Pick the metrics and their pass/fail rules before the first request reaches the canary. If you choose them after looking at the dashboards, it is too easy to explain away a real regression or to chase noise.

Measure the same signals on both groups over the same time window:

Signal What to compare Why it matters
Error rate 5xx and unhandled exceptions as a ratio of requests, per route group Catches outright breakage
Latency p50, p95, p99 per route group Tail latency often regresses before the median moves
Saturation CPU, memory, DB connection pool usage Reveals leaks and expensive queries early
Business signal Checkout or signup completion rate Catches bugs that return HTTP 200 but still break the flow

Break every metric down by route group. In a monolith, a regression on a low-volume admin route disappears inside a fleet-wide average.

Compare against a baseline that is running at the same time, not against historical data. Both groups then see the same traffic mix, the same time-of-day pattern and the same dependency incidents. If the database slows down, both groups degrade and the difference between them stays flat. Compared against last Tuesday's numbers, the same slowdown would look like a canary failure.

To make this comparison possible, tag every log line, trace span and metric with a version identifier. In practice, the structured logger adds a version field read from a build-time environment variable, and the request counters and latency histograms carry a matching version label. Use exactly one value per deployed build so label cardinality stays low.

Thresholds need tuning in both directions. If they are too tight, the canary flaps and rolls back on noise. If they are too loose, real regressions pass. Express thresholds relative to the baseline, require a minimum sample size, and only fail after the threshold is breached across several consecutive windows. To calibrate, run an A/A comparison (baseline against baseline) first and measure how much the two groups differ when nothing has changed.

import json, logging, os, time
from prometheus_client import Counter, Histogram

# One value per deployed build (e.g. git SHA set at build time) keeps label cardinality low.
VERSION = os.environ.get("APP_VERSION", "unknown")

class JsonFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({
            "ts": self.formatTime(record),
            "level": record.levelname,
            "msg": record.getMessage(),
            "version": VERSION,  # lets log queries split canary from baseline
            "route_group": getattr(record, "route_group", None),
        })

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
log = logging.getLogger("app")
log.addHandler(handler)
log.setLevel(logging.INFO)

# Same version label on metrics, so dashboards and alerts can compare groups side by side.
# route_group must be a small fixed set ("checkout", "admin", ...), never the raw URL path.
REQUESTS = Counter("http_requests_total", "HTTP requests", ["version", "route_group", "status_class"])
LATENCY = Histogram("http_request_duration_seconds", "HTTP latency", ["version", "route_group"])

def record_request(route_group, status, started):
    elapsed = time.perf_counter() - started
    REQUESTS.labels(VERSION, route_group, f"{status // 100}xx").inc()
    LATENCY.labels(VERSION, route_group).observe(elapsed)
    log.info("request completed", extra={"route_group": route_group})

A Python service that stamps the same build version onto every structured log line and onto Prometheus request counters and latency histograms, broken down by route group, so canary and baseline can be compared over the same time window.

How much traffic and soak time does a canary need to detect a regression?

The A/A run gives you the noise floor. The sample size tells you how long each step has to last before that floor stops hiding a real regression.

Illustrative estimate (assumed numbers, not a benchmark). Assume the monolith serves 20 requests/s. The baseline error rate is 0.5%, and you want to catch a rise to 1.0% at roughly 95% confidence and 80% power. Using a standard two-proportion approximation, the required requests per group are:

n ≈ (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₂ − p₁)² = 7.84 × (0.004975 + 0.0099) / 0.000025 ≈ 4,700 requests

A 1% canary receives 0.2 requests/s. A 15-minute soak at that rate is about 180 requests, so you would expect roughly one error if the canary is healthy and two if it is broken. Neither threshold logic nor a person can separate those outcomes. To collect 4,700 requests, soak time = n ÷ (total rate × canary fraction).

Stage Traffic Min requests Min soak (at 20 req/s) Gate to advance
Internal users Staff only Smoke checks pass 30 min No new error types in logs
Canary 1% 0.2 req/s 4,700 ~6.5 h Error and latency within A/A noise
Canary 10% 2 req/s 4,700 ~40 min, floor 1 h Same, plus no rise in DB load
Canary 50% 10 req/s 4,700 ~8 min, floor 1 h Same

The one-hour floors are there for time-based failures such as cache expiry, scheduled jobs and connection-pool drift. A request count alone will not surface these.

On a low-traffic service, the early steps may never reach n in any reasonable time. A green dashboard in that situation means you lack data, not that the release is safe. You can extend the soak, start at a higher percentage, or add synthetic traffic that exercises the changed code paths. Synthetic traffic only covers the paths you thought to script, though.

What does a safe rollback strategy for a monolith canary look like?

Rollback for a monolith canary happens in two steps, in a fixed order. First, set the canary's routing weight to zero so no new requests reach it. Second, redeploy or scale down the canary instances. Routing is the fast lever: it takes effect in seconds and doesn't depend on a build or image pull. Redeploying first leaves users on a failing version while the deploy runs.

Shifting traffic back is safe only if the old version can read every row the new version wrote during the canary. If the canary wrote a new enum value, a renamed column, or a changed serialization format, the stable fleet can fail on that data after rollback. This is why the expand/contract discipline from the shared-database section matters. Test it by rehearsing rollback in staging, or on a low-risk release, before you depend on it. A rollback path that has never run will likely fail when you need it.

Automated triggers should use the metrics and thresholds you defined before the deploy. If error rate or latency for the canary cohort crosses its limit, set the weight to zero without waiting for a human. Then notify someone.

Rollback cannot undo side effects. Emails already sent, payments already captured, and records written in a new format all stay in place. Each needs a compensating action: a correction email, a refund, or a backfill job that converts data to the old format.

A runbook order:

  1. Set the canary weight to 0 and confirm the stable fleet is serving all traffic.
  2. Release or expire sticky sessions pinned to the canary.
  3. Scale down the canary instances.
  4. Query for records written by the canary and run compensations.
  5. Record the trigger metric and its timeline.

When should you avoid a monolith canary, and what are the alternatives?

A canary pays off when a small slice of traffic produces enough signal to detect a regression quickly, and when a bad release would hurt many users before anyone noticed. It is a poor fit in three cases:

  • Low traffic. At a few requests per minute, a 5% slice may need hours before an error-rate difference stands out from noise.
  • Small teams with low-risk deploys and fast, cheap rollbacks. If a redeploy takes a minute and incidents are rare, the canary machinery costs more than it protects.
  • Failures that need full load or long time. Memory leaks, connection pool exhaustion at peak concurrency, and daily batch jobs can stay invisible on a small slice during a short soak.
Alternative Catches well Misses Main cost
Blue-green Startup and config failures; gives instant cutover back Load-dependent bugs, because 100% switches at once Two full environments during cutover
Feature flags with dark launch Logic errors in new code paths, toggled per user without a deploy Changes outside flagged code, such as dependency upgrades Flag cleanup and a larger combined test surface
Rolling deploy Crash loops on early instances Subtle regressions, since rollout keeps going without metric gates Mixed versions for the duration of the rollout
Pre-production load and soak testing Leaks, pool exhaustion, batch behaviour at scale Real user traffic patterns and data Realistic test environment and test data

The cost of a canary is concrete. You run a parallel canary pool, do compatibility work so both versions read and write the same data, and keep strict expand-contract migration discipline. Set that against what it saves. Incidents stay confined to a small slice of users. Detection is faster. Engineers also spend less time firefighting and compensating for bad records afterwards. If your incident history shows few regressions that a canary would have caught, put that budget into soak testing instead.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to canary deployment monolith, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.