What is a monolith deployment checklist and why does a single deployable raise release risk?
A monolith deployment checklist is a short set of gates that each release must pass before it reaches production. Every gate asks one of two questions. Is this change compatible with what is already running? Can we undo it? It is not a deploy-day ritual of announcements and sign-offs. If an item on the list does not lower the risk of incompatibility or make a rollback more likely to succeed, remove it.
A single deployable raises the stakes because one artifact carries many unrelated changes. A payments fix, a search tweak and a reporting migration from three different teams all ship as one build. If any one of them is defective, the whole release stops. Rolling back removes every team's work, including changes that were fine. The more commits a release bundles, the harder it is to find which one caused the problem, and the more likely a rollback collides with a schema change that cannot be reversed.
Smaller, more frequent releases are the most effective way to improve release safety. Each release has fewer changes to suspect and a narrower rollback plan. Each release should also have a named release owner. This is one person who decides whether to go ahead, pause or revert, so nobody has to negotiate in a group chat during an incident.
The checklist is for backend and platform engineers who ship a shared deployable. It applies to any release where old and new versions serve traffic at the same time, as in rolling or canary deploys. It also applies to any release that changes the schema, where migration ordering determines whether a rollback is possible at all.
Who should use this checklist?
The checklist is written for the people who own a release from merge to production:
- Release owners. The person who makes the go, pause or revert decision uses the checklist to know what has been verified and what the rollback plan is before the deploy starts.
- Backend engineers contributing to a shared deployable. Anyone whose change touches schema, serialized data, queue messages or shared configuration needs to know which gates their change must pass. This matters most for commits under risky paths, which must name a release contact.
- Platform and CI/CD engineers. They turn the manual items into pipeline checks, such as digest comparison, changelog generation and contact enforcement. A checklist that lives only in a document decays.
- Engineers who write and review migrations. Expand/contract ordering and lock timeouts are their part of the list.
- On-call engineers. During an incident, the changelog, named contacts and pre-agreed rollback option are what they reach for first.
It is most useful for teams where several groups merge into one artifact and no single person can read every change in a release. A small team with one monolith and a single author per release still benefits from the migration and rollback sections. The changelog and contact gates matter less there because everyone already knows what shipped.
When should you use it?
Use the full checklist for any release that has at least one of these properties:
- A mixed-version window. Rolling, canary and blue-green deploys all have a period where old and new code serve traffic or read each other's data. Compatibility checks exist for this window.
- A schema change. Any DDL, backfill or constraint change needs expand/contract ordering and a tested answer to "does the previous artifact still run on the migrated schema?"
- Changes to anything that crosses the version boundary. This includes serialized payloads, cache formats, queue messages, session formats and API contracts.
- Changes from several teams bundled together. The generated changelog and named contacts pay off when the person on call did not write most of the code.
- Changes to authentication, payments or shared configuration. These are the paths where a failure is expensive and a rollback may not undo side effects.
- Irreversible side effects. Releases that send emails, call partner APIs or overwrite data need the irreversible steps marked in the rollback plan before the deploy starts.
When should you avoid it?
The full checklist is the wrong tool in some situations, and applying it anyway adds cost without lowering risk:
- Deploys that stop all old instances before starting new ones, with an accepted downtime window. If no old and new versions ever run together, the mixed-version compatibility checks do not apply. Schema reversibility and the rollback plan still do.
- Systems with no production users yet. Prototypes and internal sandboxes where data can be thrown away do not need expand/contract migrations or go/no-go baselines. Introduce the checklist before the first release that real users or real data depend on.
- Config-only changes. A copy edit or a timeout tweak can skip migration and compatibility checks, as described in the last section. It still needs the pre-release gates, a monitored rollout and a known way to revert.
- As a substitute for tests or review. The checklist verifies compatibility and reversibility. It does not verify that the feature is correct. Do not add items that duplicate what tests already cover.
- As a growing list of manual sign-offs. If an item cannot be automated and does not change a decision, remove it rather than asking people to tick it.
Avoid splitting a change into expand/contract steps when nothing requires it. Examples include adding a new table that no existing code reads, or a change to a small table that you have confirmed takes only a brief lock. In those cases a single migration is fine, as long as the old artifact can still run against it.
How does the checklist work?
The checklist follows the release from build to cleanup. Each stage either proves compatibility or prepares the rollback, and each maps to a section below:
- Pre-release gates in CI. The pipeline checks that the build is reproducible, that the artifact being promoted is the one that was tested, that a changelog can be generated from version control, and that risky changes name a contact. Any failure blocks promotion.
- Expand migrations before the code deploy. Additive schema changes run separately from the application deploy, with lock timeouts, so both the running code and the incoming code work against the new schema.
- Deploy tolerant code with new behaviour off. The new release reads and writes both data shapes, and new features ship dark behind flags.
- Limit exposure. A canary, percentage rollout or blue-green switch limits how many users reach the new binary, while flags limit exposure to individual behaviours.
- Observe against a baseline and decide. During a fixed observation window, the release owner compares error rate, latency, saturation, queue depth and key business actions with pre-deploy values. They then choose to continue, turn off a flag, redeploy the previous artifact or roll forward, as decided in the written rollback plan.
- Verify background work, then clean up. Workers and scheduled jobs are confirmed on the new version. Contract migrations and flag removal are scheduled as owned tasks for a later release. Problems that recur across release records become new automated gates.
The order matters. The schema expands before the code that depends on it, code becomes tolerant before behaviour changes, and the schema contracts only after no old code is running. At every point, the system can move one step back or one step forward without depending on a version that is not deployed.
Which pre-release gates should every monolith release pass?
Four gates catch most of what makes a monolith release hard to reason about afterwards:
- Reproducible build. The same commit and lockfile produce the same artifact. Without this, "it passed in staging" tells you nothing about what reached production.
- Immutable artifact, promoted unchanged. Build once, identify the artifact by its content digest, and move that exact artifact through each environment. Rebuilding per environment brings back drift in dependencies, base images and build flags.
- Generated changelog. List every merged change between the deployed artifact and the candidate, derived from version control rather than written by hand. A single monolith release often bundles unrelated work from several teams, and a hand-written summary drops the small change that ends up mattering.
- Named contact per risky change. Anything touching schema, authentication, payments or shared configuration has a person who will answer during the release window.
The failure mode is box-ticking: someone marks an item done without checking it. Manual gates decay under deadline pressure, and nobody notices until an incident. Put each gate in the pipeline instead. Refuse to promote an artifact whose digest differs from the tested one, fail the release if the changelog cannot be generated, and block deploys where a risky change has no owner.
The payoff comes when something breaks. The changelog is the suspect list, the right person is already known, and immutable artifacts mean you can redeploy the previous digest or bisect between two known builds.
#!/usr/bin/env bash
set -euo pipefail
DEPLOYED_SHA="$1" # commit of the artifact currently in production
CANDIDATE_SHA="$2" # commit being released
ARTIFACT="$3" # artifact about to be promoted
REBUILD="$4" # second, independent build of CANDIDATE_SHA
TESTED_DIGEST="$5" # file holding the digest recorded when staging tests passed
fail() { echo "GATE FAILED: $*" >&2; exit 1; }
digest() { sha256sum "$1" | cut -d' ' -f1; }
[ "$(digest "$ARTIFACT")" = "$(digest "$REBUILD")" ] || fail "build is not reproducible"
[ "$(digest "$ARTIFACT")" = "$(cat "$TESTED_DIGEST")" ] || fail "digest differs from tested artifact"
git log --no-merges --format='%h %an %s' "$DEPLOYED_SHA..$CANDIDATE_SHA" > changelog.txt
[ -s changelog.txt ] || fail "changelog is empty or could not be generated"
RISKY='^(db/migrations|app/auth|app/payments|config/shared)/'
missing=0
for c in $(git rev-list --no-merges "$DEPLOYED_SHA..$CANDIDATE_SHA"); do
if git diff-tree --no-commit-id --name-only -r "$c" | grep -E "$RISKY" > /dev/null; then
if ! git log -1 --format=%B "$c" | grep '^Release-Contact: ' > /dev/null; then
echo "risky commit without contact: $(git log -1 --format='%h %s' "$c")" >&2
missing=1
fi
fi
done
[ "$missing" -eq 0 ] || fail "risky changes lack a named contact"
echo "all gates passed for $(digest "$ARTIFACT")"
A CI gate script that blocks promotion unless the build is reproducible, the artifact digest matches the tested one, a changelog can be generated from git, and every commit touching risky paths names a release contact.
How does expand/contract migration ordering keep schema and code compatible?
Expand/contract splits one schema change into steps that are each safe against both the running code and the code about to replace it.
- Expand. Add nullable columns and new tables only. Nothing existing breaks, because old code never reads them.
- Deploy tolerant code. The new release writes both shapes and reads the new one with a fallback to the old.
- Backfill outside the deploy. Run it as a separate job in small batches, so one slow update can't stall a release or hold long locks.
- Contract. In a later release, after no instance of the old code is running, add
NOT NULLor foreign keys and drop the old column.
The risk sits in the DDL itself. Some column changes rewrite the whole table. A plain index build can block writes. Validating a constraint on a large table scans every row. The worse failure is a DDL statement queued behind a long transaction, which then blocks every query that arrives after it.
Set a short lock timeout before each step (in PostgreSQL, SET lock_timeout = '3s') so the migration fails fast and can be retried. For example, expand with ALTER TABLE orders ADD COLUMN region text. Contract later by adding a constraint as NOT VALID, then running VALIDATE CONSTRAINT separately, which takes a weaker lock.
The cost is real:
- One change becomes two or three releases.
- Dual-path code lingers until someone removes it.
- Decoupling schema from code creates version drift. Test new code against the old schema and old code against the new schema, because either combination can be live.
-- PostgreSQL. Each step runs separately, never inside the app deploy.
-- EXPAND (release N): metadata-only change, safe for old and new code.
SET lock_timeout = '3s'; -- fail fast instead of queueing behind a long transaction
ALTER TABLE orders ADD COLUMN region text; -- nullable, no default: no table rewrite
-- Build the index without blocking writes (cannot run inside a transaction block).
CREATE INDEX CONCURRENTLY IF NOT EXISTS orders_region_idx ON orders (region);
-- BACKFILL (separate job): repeat until 0 rows are updated.
SET statement_timeout = '5s'; -- keep each batch short
UPDATE orders
SET region = COALESCE(legacy_region, 'unknown')
WHERE id IN (
SELECT id FROM orders WHERE region IS NULL LIMIT 1000
);
-- CONTRACT (release N+1 or later, after no old code is running).
SET lock_timeout = '3s';
-- Add the rule without scanning existing rows; new writes are checked immediately.
ALTER TABLE orders
ADD CONSTRAINT orders_region_not_null CHECK (region IS NOT NULL) NOT VALID;
-- Scan existing rows under a weaker lock that allows reads and writes.
ALTER TABLE orders VALIDATE CONSTRAINT orders_region_not_null;
-- On PostgreSQL 12+, the validated CHECK lets SET NOT NULL skip the full scan.
ALTER TABLE orders ALTER COLUMN region SET NOT NULL;
ALTER TABLE orders DROP CONSTRAINT orders_region_not_null;
-- Remove the old path only once nothing reads it.
ALTER TABLE orders DROP COLUMN legacy_region;
A PostgreSQL expand/backfill/contract sequence that uses lock and statement timeouts, a concurrent index build, and a NOT VALID constraint validated separately, so no step holds a long blocking lock.
What must stay readable when old and new versions run at the same time?
During a rolling deploy, old and new instances serve traffic side by side for minutes. In a blue-green deploy, the overlap is shorter, but it still exists: in-flight requests, queued jobs and cached data written by blue are read by green. If you roll back, the roles reverse. Every artifact that crosses the version boundary must be readable in both directions. A new reader must accept old data, and an old reader must tolerate new data, usually by ignoring unknown fields rather than rejecting them.
Background workers are the most common blind spot. Queue consumers and cron jobs often deploy on a separate schedule, from a separate pipeline, or only when someone remembers to restart them. A web instance on the new version can enqueue a message that a worker still on the old version cannot parse. That failure lands in a dead-letter queue or a log line, not in the HTTP health check. The load balancer stays green while jobs silently fail. Track worker versions explicitly and alert on consumer error rates, not just web-tier status.
| Artifact | Risk in a mixed-version window | Compatibility check |
|---|---|---|
| Serialized payloads (JSON, binary formats) | Old code rejects new fields or a changed type | Deserialize new samples with old code and old samples with new code |
| Cache keys and values | New code reads an old-format value and crashes | Version the key prefix, or confirm both versions can parse both value formats |
| Queue messages | Old workers drop or dead-letter new messages | Add fields optionally, and deploy consumers before producers |
| Session formats | Users get logged out or errors when routed between versions | Confirm both versions can decode sessions written by either |
| API contracts | Clients or internal callers break on removed or renamed fields | Make additive changes only, and run contract tests against both versions |
How do feature flags compare with release trains, canaries and blue-green deploys?
These techniques limit how many users a bad release reaches, and they combine rather than compete. A flag separates deploying code from exposing behaviour. A traffic strategy limits who receives the new binary.
| Approach | What it limits | Main cost | Common failure |
|---|---|---|---|
| Feature flags | Exposure of one behaviour | Code paths that must be maintained | Untested flag combinations, stale flags |
| Release trains | How often you deploy | Larger batches | Hard to isolate the change that broke |
| Frequent small releases | Size of each change | Pipeline speed and test reliability | Little, if the pipeline is fast |
| Canary / percentage traffic | Share of users on the new version | Routing and spare capacity | Low-traffic paths never exercised |
| Blue-green | Time to switch back | A full second environment | Shared database still not reversible |
Ship flagged code dark and turn it on separately. Give each flag an owner and a removal date. Every live flag doubles the number of states the code can be in, and nobody tests most of them.
A canary sees only what its traffic sample contains. A nightly batch job or a rarely used admin endpoint can look healthy for hours and then fail. Warm-up also skews early signals. Cold connection pools, empty caches and an unwarmed JIT make new instances look slower than they really are. Readiness checks should confirm real dependencies, such as the database and the queue, not just that the process is running.
For a monolith with high memory use and slow startup, blue-green can mean paying for twice the capacity, and canaries can make each deploy take much longer. In that case, small frequent releases with flags are usually cheaper.
How do you write a rollback plan that will actually work?
A rollback plan is a decision made before the deploy, not during the incident. For each release, write down which of three moves you will use and when.
| Option | Undoes | Does not undo | Use when |
|---|---|---|---|
| Redeploy previous artifact | Code behaviour | Schema changes, rewritten rows, sent messages | Bug is in code and the old build runs on the current schema |
| Turn off a flag | One feature path | Data already written by that path | The change was shipped dark and is isolated behind the flag |
| Roll forward | Nothing; adds a fix | — | Data or schema has moved past what the old build understands |
Redeploying only works if the previous artifact still runs against the post-migration schema. Test this in staging by deploying the old build onto the migrated database. Do not assume it works because the expand step "should" be compatible. If the old build fails there, your only rollback paths are the flag and rolling forward.
Mark every irreversible step in the plan, such as backfills that overwrite columns, calls to payment or partner APIs, and emails or webhooks already sent. Rolling back the artifact changes which code runs. It does not restore rows or recall a message. Once those steps run, rolling forward with a corrective fix is often the only realistic option, so have someone ready to write it.
Agree on go/no-go criteria in advance and compare each against a pre-deploy baseline:
- Error rate
- p95/p99 latency
- CPU, memory and connection-pool saturation
- Queue depth
- Completion of key business actions such as checkouts or sign-ups
Set a fixed observation window. Name one person who can halt the release without asking for consensus.
What should happen after the release to keep the next one cheaper?
Once the observation window closes, check the work that does not run on request paths. Background jobs and scheduled tasks often pick up new code only when their workers restart, and a nightly job may not run until hours after the release looks healthy. Confirm that each job type has completed at least once on the new version, that retry queues are not growing, and that no worker is still running the old build.
Then write a short release record: what shipped, what deviated from the plan, which alerts fired, and how long each step took. Keep it brief enough that people actually fill it in. Over several releases these records show where time goes.
Before closing the release ticket, schedule the follow-up work as named tasks with owners and dates. This includes the contract-phase migration that drops old columns and the removal of flags that are now fully on. Cleanup debt that is tracked stays small. Untracked debt builds up until nobody is sure which columns are still read or which flag branches are dead, and every later release has to work around them.
When the same problem appears in two release records, turn it into an automated gate. Examples include a migration that locked a large table or a missing default on a new column. A lint rule or CI check costs a few hours to build once. A manual review step costs reviewer time on every release.
Not every release needs the full list. A config-only change, such as a copy edit or a timeout tweak, can skip migration and compatibility checks. It still needs the pre-release gates, a monitored rollout, and a known way to revert.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to monolith deployment checklist, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.