What are feature toggles in a monolith, and why separate deploy from release?

A feature toggle is a runtime switch that decides whether a code path executes. Its job in a monolith is to split two events that usually happen together. Deployment puts new code into the single artifact and onto production machines. Release exposes that code to users. With a toggle in place, a half-built module can ship in every deploy, dormant and unreachable, until someone flips the switch. That decision is made through configuration, not a new build.

This matters most under trunk based development. In a monolith, every team commits to one branch and ships one artifact. If unfinished features have to wait on long-lived branches, merges get large, conflicts pile up, and the deploy train stalls behind the slowest team. Toggles remove that wait. Engineers merge small changes to trunk daily, the pipeline deploys continuously, and incomplete work stays switched off. One team's partial feature never blocks another team's fix from reaching production.

This guide takes a specific angle. A toggle should be a temporary, owned seam at a module boundary: one decision point where a request is routed to the old implementation or the new one. It should have a named owner and a planned removal date. The failure mode it avoids is the same flag checked in a dozen if statements across controllers, services and templates. That pattern is hard to reason about, hard to test, and rarely gets deleted.

The teams that benefit most are backend and platform groups maintaining a shared monolith that many teams commit to. That is where coupling between deploy and release costs the most.

Which types of feature toggles do you need, and how long should each live?

Toggles that look alike in code serve different purposes, and each purpose implies a different lifespan and owner. Classify every toggle when you create it, because the classification decides when it gets deleted.

Type Expected lifespan Safe default Owner Evaluation Removal rule
Release Days to a few weeks Off Feature team Mostly static (per deploy or config) Delete once the feature is fully on in production
Ops / kill switch Months to indefinite On (feature running) Platform or on-call Dynamic, changeable at runtime Review periodically; remove when the dependency it guards is gone
Experiment One experiment cycle Control variant Product plus the owning team Dynamic, per request or user Delete when a winner is chosen and the result is recorded
Permission As long as the entitlement exists Deny Product or account management Dynamic, per user or tenant Treat as product logic; move it into your authorization model

The lifespans differ for a reason. A kill switch is insurance. It lets on-call shed a slow recommendation module or a flaky third-party call without a deploy, and that capability stays valuable for as long as the risk exists. A release toggle has no lasting value. Every week it survives, the codebase carries two paths that both need testing, and the "off" path gradually stops working without anyone noticing.

Mixing purposes is how short-lived toggles become permanent. Suppose a release toggle also starts gating a premium tier. The feature team can no longer delete it because product now depends on it, and nobody owns it clearly. Use one purpose per toggle, give it a named owner, and set an expiry date.

How does dark launching a module work inside one process?

Dark launching depends on one rule: the choice between old and new code happens in exactly one place. Put that dispatch point at the module's entry, which might be the router, a service factory or the dependency-injection binding. Both implementations sit behind the same interface. Callers never check the flag themselves. If toggle checks spread into call sites, removing the toggle later means a search through the whole codebase. With one dispatch point, removal is a one-line change.

request
   |
   v
[dispatch point] --flag/cohort--> live path (old or new) --> response
   |
   +--shadow enabled--> async queue --> shadow path (new, bounded)
                                            |
                                            v
                              comparison / logging sink --> discard

Shadow execution runs the new module next to the old one while users only see the old result. The shadow call must stay off the request's critical path:

  • Hand it to a bounded queue or worker pool.
  • Give it a timeout and a concurrency cap.
  • Drop work when the queue fills instead of making callers wait.

If you skip these limits, a slow new implementation adds latency to every request, even though users never see its output. The sink compares the new output with the live result, records any differences and throws the shadow output away.

Percentage and cohort rollouts hash a stable identifier, such as a user ID or tenant ID, into a bucket. Don't use a random draw for each request. Hashing gives a user the same path on every request and on every instance, which keeps sessions coherent and makes bug reports reproducible.

                 +--------------------+
                 | request (user ID)  |
                 +---------+----------+
                           v
     +---------------------+---------------------+
     | dispatch point: router / factory / DI     |
     | bucket = hash(stable ID) -> same path on  |
     | every request and every instance          |
     +-------+---------------------------+-------+
             |                           |
             | flag/cohort               | shadow enabled
             v                           v (non-blocking)
     +-------+---------+       +---------+--------------+
     | live path       |       | bounded queue          |
     | old OR new impl |       | full -> drop, no wait  |
     | shared iface    |       +---------+--------------+
     +--+---------+----+                 v
        v         |            +---------+--------------+
   +----+-----+   |            | shadow path (new impl) |
   | response |   |            | timeout, concurrency   |
   | to user  |   |            | cap, off critical path |
   +----------+   |            +---------+--------------+
                  |                      v
                  |            +---------+--------------+
                  |            | comparison / log sink  |
                  +--live----->| record diffs, then     |
                    result     | discard shadow output  |
                               +------------------------+

A single dispatch point hashes a stable ID to pick the live implementation and, when shadowing is on, hands a copy of the request to a bounded queue so the new module's output is compared against the live result in the sink, logged and discarded without delaying the response.

How do you implement a toggle at a module seam?

Start by defining the module behind an interface, such as PricingEngine, with two implementations: the current one and the candidate. Callers depend only on the interface, so they cannot tell which implementation they receive. A factory or DI binding chooses the implementation. Static toggles are read once at wiring time, so request code contains no conditionals.

For percentage rollout, the factory hashes a stable ID together with the toggle name and takes the result modulo 100. Including the toggle name salts the hash. Without it, the same users would land in the early cohort of every rollout you run.

For dark launch, bind a shadow wrapper. It serves the current implementation's response, runs the candidate asynchronously with a hard timeout, and compares the two outputs. The candidate gets a no-op mailer, a read-only repository, and a discarding event publisher. Its failures are logged and never propagated.

Evaluation design

  • Static vs. dynamic. Read kill switches and wiring choices at startup. Use dynamic, per-request values only for rollouts that must change without a restart.
  • Safe defaults. If the config source is unreachable, fall back to the last known value, then to the declared default. The default should always be the old path.
  • Hot-path cost. Keep a local in-memory cache that refreshes in the background. Evaluation then costs a map lookup, not a network call.
  • Consistency across instances. Deterministic hashing keeps a user's path stable. Version the config so you can see which instances have picked up a change.

Each toggle definition should carry:

  • name: pricing-v2-shadow
  • owner: payments-team
  • type: release
  • default: off
  • expiry: a date that fails CI once it has passed
import { createHash } from "node:crypto";

interface PricingEngine { quote(cartId: string, userId: string): Promise<number>; }
type PricingToggle = { mode: "off" | "shadow" | "rollout"; percent: number }; // default: off

// Deterministic 0-99 bucket. Salting with the toggle name gives each rollout its own cohort.
function bucket(toggle: string, stableId: string): number {
  return createHash("sha256").update(`${toggle}:${stableId}`).digest().readUInt32BE(0) % 100;
}

// Serves current, runs candidate off the response path. The candidate must be built
// with a no-op mailer, read-only repository and discarding event publisher.
class ShadowPricing implements PricingEngine {
  constructor(private current: PricingEngine, private candidate: PricingEngine, private timeoutMs = 200) {}
  async quote(cartId: string, userId: string): Promise<number> {
    const served = await this.current.quote(cartId, userId);
    void this.compare(cartId, userId, served); // not awaited, so it cannot add latency
    return served;
  }
  private async compare(cartId: string, userId: string, served: number): Promise<void> {
    let timer: ReturnType<typeof setTimeout> | undefined;
    const timeout = new Promise<never>((_, reject) => {
      timer = setTimeout(() => reject(new Error("shadow timeout")), this.timeoutMs);
    });
    try {
      const shadow = await Promise.race([this.candidate.quote(cartId, userId), timeout]);
      if (shadow !== served) console.warn("pricing-v2 mismatch", { cartId, served, shadow });
    } catch (err) {
      console.warn("pricing-v2 shadow failed", err); // log only, never propagate
    } finally { clearTimeout(timer); }
  }
}

// Wiring: the toggle value is read here, so request code has no conditionals.
function choosePricing(t: PricingToggle, userId: string, current: PricingEngine,
                       candidate: PricingEngine, shadow: ShadowPricing): PricingEngine {
  if (t.mode === "shadow") return shadow;
  if (t.mode === "rollout" && bucket("pricing-v2", userId) < t.percent) return candidate;
  return current; // "off", unknown modes and missing config all fall back to the old path
}

A TypeScript module seam with a salted, deterministic rollout bucket, a shadow wrapper that runs the candidate with a timeout and only logs its result, and a factory that defaults to the current implementation.

Shared state, side effects, testing and observability

A toggle only isolates code paths. The database, the message queues and the outside world are still shared, so those need their own safeguards.

Shared state

Both toggle states run against one schema, so migrations must work for both. Use expand-then-contract. First add new columns or tables without breaking old readers. Then let both paths run. Drop the old structure only after the toggle is removed. A migration that locks a hot table or renames a column hurts every user, whatever their toggle state. The same goes for dark code that writes. A shadow path that updates real rows is a production change. Point it at separate tables, or keep it read-only until release.

Side effects

Dark code must not send emails, charge cards or call partner APIs. Route these calls through an interface that the toggle can switch to a stub or a recording sink. This lets you inspect what would have been sent without anything leaving the system.

Testing

Run CI with each active toggle both on and off. You do not need every combination. If toggles stay independent, with no toggle checking another, one job per toggle state is enough. For example, the matrix could have pricing-v2-shadow=off, pricing-v2-shadow=on, and a "defaults" job. Release-candidate checks must include that defaults job, because the default state is what production gets.

Observability

Attach toggle state to logs, metrics and trace spans, for example toggle.pricing-v2-shadow=on. You can then compare error rates and p99 latency between the two paths directly. If the new path is worse, flip the toggle off to roll back. No redeploy is needed.

# GitHub Actions workflow: one job per toggle state, plus a defaults job.
# Assumes the app reads FEATURE_TOGGLE_OVERRIDES (comma-separated name=state)
# at startup. Any toggle not listed keeps its default value.
name: ci
on: [push, pull_request]

jobs:
  test:
    name: test (${{ matrix.label }})
    runs-on: ubuntu-latest
    strategy:
      # Run every state even if one fails, so you can see which path broke.
      fail-fast: false
      matrix:
        include:
          # No overrides. This is exactly what production runs, so
          # release-candidate checks must require this job to pass.
          - label: defaults
            overrides: ""
          # Each job overrides one toggle and leaves the rest at their defaults.
          # This only works because toggles never check each other.
          - label: pricing-v2-shadow=off
            overrides: "pricing-v2-shadow=off"
          - label: pricing-v2-shadow=on
            overrides: "pricing-v2-shadow=on"
          - label: invoice-export=off
            overrides: "invoice-export=off"
          - label: invoice-export=on
            overrides: "invoice-export=on"
    steps:
      - uses: actions/checkout@v4
      - name: Run test suite
        env:
          FEATURE_TOGGLE_OVERRIDES: ${{ matrix.overrides }}
          # Dark code must not reach real providers in CI either.
          EMAIL_TRANSPORT: recording-sink
          PAYMENT_GATEWAY: stub
        run: make test

A GitHub Actions matrix with one job per toggle state plus a defaults job. Because toggles are independent, the job count grows linearly with the number of toggles, not with every combination.

When should you avoid feature toggles in a monolith, and what are the alternatives?

That rollback is fast, but it has costs. Each toggle doubles the reachable states, and toggles interact: ten booleans produce 1,024 combinations, and you will test only a few. Shadow execution spends live CPU, memory and database connections on work no user sees. Dynamic evaluation makes the flag store a runtime dependency, and caching flags means two instances can briefly disagree. Dark code shares the process, so a crash, leak or exhausted thread pool takes down the live path too. A flip changes production behaviour without a pull request, so it needs its own access control and change log. Integration work you skipped also meets real traffic for the first time.

Avoid toggles in three cases:

  • Risky schema or data changes. A toggle cannot un-write rows. Use expand-and-contract migrations instead.
  • Modules that need resource or fault isolation. One process means one blast radius.
  • Tiny changes. If a revert costs less than the toggle's lifecycle, skip the toggle and revert if needed.
Approach Isolation Feedback speed Cost
Runtime toggle None (shared process and data) Fast, real traffic Toggle debt, combinations to test
Long-lived feature branch Full until merge Slow, merge pain Drift and late integration
Branch by abstraction, no runtime toggle None at runtime Fast to integrate, released by deploy Abstraction layer upkeep
Canary or blue-green deploy Per instance Moderate Extra capacity, routing
Separate service Process and resources Slow at first Network calls, ops overhead

How do toggles affect development cost, and how do you keep them from becoming debt?

Toggles carry real costs. You pay up front to cut the seam: an interface, a second code path, and wiring to the evaluation call. You pay on every CI run to exercise both states, and you pay to run the evaluation infrastructure. Later you pay again to remove the toggle. In return, you avoid rebasing a long-lived branch, you skip a dedicated release deploy, and a bad rollout becomes a configuration change instead of an emergency hotfix.

Illustrative estimate (assumed numbers, not a benchmark).

  • Cost = seam work + cleanup + (extra CI minutes per run × runs)
  • Savings = merge-conflict time avoided + release-deploy time avoided + (hotfix time × probability of needing one)

Assume 8 hours of seam work, 3 hours of cleanup, and 4 extra CI minutes across 200 runs (about 13 machine-hours, but no engineer time). On the other side, assume a four-week branch costing 3 hours of conflict resolution per week, a 2-hour release deploy, and a 50% chance of a 6-hour hotfix. That gives 11 engineer-hours spent against 12 + 2 + 3 = 17 saved. Plug in your own figures. If the change is small and short-lived, the comparison often goes the other way.

The savings only hold if toggles die on schedule. Use these rules:

  • Set an owner and an expiry date at creation. Store them in the toggle definition, not in a ticket that drifts away from the code.
  • Fail CI on expired toggles. A check that breaks the build forces a decision: remove the toggle or consciously extend it.
  • Cap active release toggles per team or module. Each live toggle doubles the states someone has to reason about.
  • Remove the dead branch and the toggle in one change. Leaving the old path "just in case" leaves untested code in place without a working switch to reach it.
#!/usr/bin/env python3
"""CI gate: fail the build on expired or ownerless toggles, or too many release toggles."""
import json
import sys
from datetime import date

MAX_RELEASE_TOGGLES = 5  # per-module cap; each live toggle doubles the states to reason about


def main(path):
    # toggles.json lives next to the code: [{"name", "owner", "type", "expires": "YYYY-MM-DD"}]
    with open(path) as f:
        toggles = json.load(f)
    today = date.today()
    errors = []
    for t in toggles:
        name = t.get("name", "<unnamed>")
        # Owner and expiry are mandatory when the toggle is created, not added later
        if not t.get("owner") or not t.get("expires"):
            errors.append(f"{name}: owner and expires are required")
            continue
        expires = date.fromisoformat(t["expires"])
        if expires < today:
            # Force a decision: delete toggle + dead branch, or extend expires on purpose
            errors.append(f"{name}: expired {expires} (owner: {t['owner']})")
    release = [t for t in toggles if t.get("type") == "release"]
    if len(release) > MAX_RELEASE_TOGGLES:
        errors.append(f"{len(release)} active release toggles exceeds cap of {MAX_RELEASE_TOGGLES}")
    for e in errors:
        print(f"TOGGLE CHECK: {e}", file=sys.stderr)
    return 1 if errors else 0  # non-zero exit fails the CI job


if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "toggles.json"))

A standard-library Python script for CI that reads toggle definitions stored with the code and fails the build when a toggle has no owner or expiry, is past its expiry date, or pushes the number of active release toggles over a set cap.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to feature toggles monolith, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.