What are feature toggles in trunk based development, and why does a monolith need them?

A feature toggle is a conditional in the code that chooses between two paths at runtime. In trunk based development, feature toggles let you merge code that is not finished yet. The new path ships to production but stays switched off until someone decides to expose it. This separates deploy from release. Deploying puts a build on servers. Releasing makes a behaviour visible to users. Without a toggle, those are the same event. With one, you can deploy many times before you release once, and you can undo a release without a redeploy.

The teams that benefit most share one main branch in a single monolith. Every merge lands in the same codebase, the same build and usually the same deployment. On a long-lived feature branch, unfinished work drifts away from main for weeks. That leads to large, risky merges and conflicts in files other teams have since rewritten. A toggle removes the reason to keep that branch alive. Incomplete code merges daily in a dormant state, so each merge stays small. Conflicts surface while they are still a few lines wide, and everyone builds against the real current state of the system.

Every toggle starts with three things: a type, a named owner and a removal date. A toggle is a short-lived branch that lives inside the code. It is not a permanent configuration setting. Once the release is finished, the toggle and its losing code path should be deleted.

Which types of toggles are there, and how long should each one live?

Not every toggle should live the same amount of time. Short-lived flags turn into permanent ones when all toggles get the same treatment. Record the category when you create a toggle, because the category determines how long it lives, who reviews it and what happens when evaluation fails.

Type Purpose Expected lifespan Failure default Owner
Release toggle Hide unfinished work on trunk Days to weeks Off (old path) Feature team
Experiment toggle Route user cohorts to variants Length of the experiment Control variant Team running the experiment
Ops toggle / kill switch Shed load, bypass a failing dependency Long-lived Safe, degraded path Service owner / on-call
Permission toggle Gate features by plan or role Indefinite Deny Product, ideally as entitlement logic

Release toggles should be reviewed every iteration. If one is past its removal date, treat it as a defect.

Experiment toggles end when a decision is made, not when someone remembers to remove them. Bucketing has to stay stable, or the results are meaningless.

Ops toggles need periodic drills. A kill switch that nobody has flipped in a year may not work when you need it.

Permission toggles carry real security risk. If paid or admin-only behavior sits behind a general-purpose flag, a typo in an override, a wrong environment file or a "default on" fallback can expose that behavior to every user. Entitlements belong in the authorization and product model, where they get access-control review and audit trails. Keep them out of a flag store that many engineers can edit.

How do toggles work in a monolith? Placing the decision point and designing the evaluation layer

The toggles themselves also need a home in the code, and the most important choice is where the decision gets made. Evaluate each toggle once, at a seam: the controller, the service entry point, or the dependency-injection wiring that assembles the object graph. If if (newPricing) checks spread through domain code, every branch doubles the paths to test, and removing the toggle later means hunting through files nobody remembers touching.

A cleaner pattern is to define an interface, keep the old and new behavior as two implementations, and let the seam pick one. Domain code calls the interface without knowing a toggle exists. Inline if/else is acceptable for a short-lived release toggle that touches one spot. Once the same check shows up in a second place, move it to the seam.

Route every check through a single internal evaluation interface that takes a toggle name and a context (user, tenant, request). This gives you one place to set defaults, log decisions and swap providers. Defaults depend on the toggle type:

  • Release toggles fail closed. If evaluation errors, users get the old path.
  • Kill switches default to the safe behavior. That is usually the degraded mode, not the full feature.

On hot paths, evaluate once per request and carry the result on the request context. This keeps the cost bounded and stops a mid-request flip from producing half-old, half-new behavior.

A centralized toggle service is also a runtime dependency. If it goes down or its cache goes stale, production behavior changes without a deploy. Monitor it the same way you monitor your database.

// One internal evaluation interface: name + context in, boolean out.
type ToggleContext = { userId: string; tenantId: string; requestId: string };
interface ToggleProvider { isOn(name: string, ctx: ToggleContext): Promise<boolean>; }

// Defaults live here, keyed by toggle type, not at each call site.
const SAFE_DEFAULTS: Record<string, boolean> = {
  'release.new-pricing': false,     // release toggle: fail closed, old path
  'kill.recommendations': true,     // kill switch: true = degraded mode on
};

export async function evaluate(p: ToggleProvider, name: string, ctx: ToggleContext) {
  try {
    return await p.isOn(name, ctx);
  } catch (err) {
    console.warn(`toggle ${name} failed, using default`, err); // log the fallback
    return SAFE_DEFAULTS[name] ?? false;
  }
}

// Domain code depends only on this interface; it never sees a toggle.
interface PricingService { quote(sku: string): number; }
class LegacyPricing implements PricingService { quote(sku: string) { return 100; } }
class TieredPricing implements PricingService { quote(sku: string) { return 90; } }

// The seam: evaluate once per request, then pick the implementation.
// Every later call in this request uses the same choice, so a flip
// mid-request cannot mix old and new behavior.
export async function buildRequestScope(p: ToggleProvider, ctx: ToggleContext) {
  const newPricing = await evaluate(p, 'release.new-pricing', ctx);
  const pricing: PricingService = newPricing ? new TieredPricing() : new LegacyPricing();
  return { ctx, toggles: { newPricing }, pricing };
}

A single evaluation function applies per-toggle-type safe defaults when the provider fails, and a request-scoped seam evaluates the release toggle once and selects a PricingService implementation so that domain code never branches on the toggle.

How do you run dark launches and schema changes behind toggles safely?

A dark launch runs the new code path in production alongside the old one, but only the old path's result reaches the user. The request handler calls both, returns the old result, and logs any mismatch with enough context to reproduce it. This tests the new query or computation against real data and real traffic patterns, which staging rarely reproduces.

The new path must be read-only in effect. Wrap anything with side effects (writes, emails, webhooks, payment or third-party API calls) in a check that turns it into a no-op or a recorded intent while in shadow mode. One missed side effect can send a customer two invoices or write data twice. Run the shadow call asynchronously, or with a strict timeout, so its failures and latency never touch the response.

In a monolith, both paths usually hit the same database. Shadowing every request can double the query load on the hottest tables. Sample a small percentage of traffic, run shadow queries against a read replica where you can, and watch connection pool usage, lock waits and replication lag before you raise the rate.

Schema changes follow expand/contract. First, add new columns or tables without removing anything. Next, write to both old and new structures and backfill. Then move reads behind the toggle. Drop the old structures only after the toggle is removed. Keep the toggle's role clear: switching it off only changes which code runs. It does not restore a dropped column or reverse rows the new path has already written. Plan data rollback separately.

import logging
import random
from concurrent.futures import ThreadPoolExecutor

log = logging.getLogger("dark_launch")
# Small, bounded pool: limits how much extra load shadow queries can add at once.
_shadow_pool = ThreadPoolExecutor(max_workers=2)
SHADOW_SAMPLE_RATE = 0.02  # 2% of requests; raise only after checking pool usage, lock waits, replica lag

class SideEffects:
    """Every write, email, webhook or external API call goes through this gate."""
    def __init__(self, shadow):
        self.shadow = shadow
        self.intents = []

    def perform(self, name, fn, *args):
        if self.shadow:
            self.intents.append((name, args))  # record intent, never execute
            return None
        return fn(*args)

def invoice_total(order_id, primary_db, replica_db):
    # old_total / new_total are the app's existing and candidate implementations.
    result = old_total(primary_db, order_id, SideEffects(shadow=False))
    if random.random() < SHADOW_SAMPLE_RATE:
        # Run off the request thread so shadow latency and errors never reach the user.
        _shadow_pool.submit(_shadow_compare, order_id, replica_db, result)
    return result  # only the old path's result is returned

def _shadow_compare(order_id, replica_db, expected):
    effects = SideEffects(shadow=True)
    try:
        actual = new_total(replica_db, order_id, effects)  # read replica, not primary
    except Exception:
        log.exception("shadow failed order_id=%s", order_id)
        return
    if actual != expected:
        # Log enough context to reproduce the mismatch offline.
        log.warning("shadow mismatch order_id=%s old=%r new=%r intents=%r",
                    order_id, expected, actual, effects.intents)

A sampled dark launch that runs the new computation asynchronously against a read replica, routes all side effects through a gate that only records them in shadow mode, logs mismatches, and always returns the old path's result.

How should you test and roll out toggled code?

Testing both paths

Every active toggle creates two versions of the system, and both of them can reach production. Run the affected tests with the toggle on and with it off. A test that only covers the default state will miss the regression that appears on the day someone flips it.

Don't try to test every combination. Ten independent toggles produce over a thousand states. Keep the number of toggles that touch the same code path small, and test the combinations that can actually occur together. Set toggle values explicitly in test fixtures rather than inheriting them from shared configuration. Otherwise, changing a production default can silently change what your suite is testing.

Add a CI job that runs the suite with in-progress toggles forced on. This keeps the unfinished path green while it is still dark, so enabling it later is not the first time anyone has run it end to end.

Review toggled code with the same rigour as any other change. "It's behind a flag" is not a safety argument. The code ships in the same binary, shares the same process and database connections, and can still break things through imports, migrations, or shared state.

Rolling out gradually

Enable the toggle for internal users first, then for a small percentage of users or tenants. Bucket on a stable identifier, such as a hash of the tenant ID, so each user stays in the same group across requests. Without stable bucketing, a user can see behaviour flip between page loads.

Tag metrics, traces, and logs with the toggle state. When error rates rise, you can then tell which path produced them instead of guessing.

import hashlib
from decimal import Decimal, ROUND_HALF_UP

import pytest


def in_rollout(toggle: str, tenant_id: str, percent: int) -> bool:
    # Hash toggle name + tenant ID so a tenant always lands in the same bucket,
    # and different toggles don't all enable for the same 10% of tenants.
    digest = hashlib.sha256(f"{toggle}:{tenant_id}".encode()).digest()
    return int.from_bytes(digest[:4], "big") % 100 < percent


def invoice_total(lines: list[float], flags: dict) -> float:
    if flags["new_tax_engine"]:  # in-progress path, still dark in production
        subtotal = sum(Decimal(str(x)) for x in lines)
        return float((subtotal * Decimal("1.2")).quantize(Decimal("0.01"), ROUND_HALF_UP))
    return round(sum(lines) * 1.2, 2)


# Set toggle values explicitly per test instead of reading shared config,
# so changing a production default can't silently change what is tested.
@pytest.fixture(params=[False, True], ids=["tax_engine_off", "tax_engine_on"])
def flags(request):
    return {"new_tax_engine": request.param}


def test_invoice_total_same_on_both_paths(flags):
    # Runs twice: once per toggle state.
    assert invoice_total([10.0, 5.5], flags) == 18.6


def test_bucketing_is_stable():
    # Same tenant, same answer on every request: no behaviour flipping.
    results = {in_rollout("new_tax_engine", "tenant-4821", 10) for _ in range(50)}
    assert len(results) == 1

A pytest fixture parametrized over both toggle states with values set explicitly, plus a hash-based rollout check that keeps each tenant in a stable bucket.

How do you manage toggle debt before it becomes permanent configuration?

A release toggle that sits at 100% for months stops being a release mechanism and becomes configuration that nobody decided to keep. Other code starts depending on its state, operators flip it during incidents, and the losing branch goes untested. By the time someone tries to remove it, nobody remembers what the "off" path does, so deleting it feels risky.

To prevent this, require three fields on every toggle definition: owner, creation date, and expected removal date. Reviewers should reject any toggle that lacks them. Add a CI lint that reads the toggle registry and fails the build when a release toggle passes its removal date. The team then has to delete the toggle or approve an extension in review. Ops and permission toggles can have longer limits or none, because they are designed to persist.

Make removal part of the feature's definition of done. The ticket closes when the toggle is gone, not when it reaches 100%.

Removal checklist

  1. Confirm production state. Verify the toggle is fully on or fully off in every environment and has stayed that way.
  2. Delete the losing branch. Inline the winning path and remove the conditional.
  3. Remove the config entry. Delete it from the registry along with any per-environment overrides.
  4. Delete the tests for the dead path.

What toggles cost

Toggles move complexity from merge time to runtime and double the code paths you have to reason about. In return, they save time on merge conflicts, long-lived branch integration, and redeploy-based rollbacks.

Illustrative estimate: assume a two-week branch costs one day of integration work, and a toggle costs one hour to add and one hour to remove. Under those assumptions, the toggle comes out ahead only if the removal actually happens.

#!/usr/bin/env python3
# CI lint: fail the build if any toggle lacks ownership metadata
# or a release toggle is past its expected removal date.
# Usage: python3 lint_toggles.py config/toggles.json
import json
import sys
from datetime import date

REQUIRED = ("owner", "created", "remove_by")

def main(path):
    with open(path) as f:
        toggles = json.load(f)  # list of {name, type, owner, created, remove_by}

    today = date.today()
    errors = []
    for t in toggles:
        name = t.get("name", "<unnamed>")
        missing = [k for k in REQUIRED if not t.get(k)]
        # Ops and permission toggles are meant to persist; only release toggles expire.
        if t.get("type") != "release":
            continue
        if missing:
            errors.append(f"{name}: missing {', '.join(missing)}")
            continue
        if date.fromisoformat(t["remove_by"]) < today:
            errors.append(
                f"{name}: expired {t['remove_by']} (owner: {t['owner']}) "
                "- remove it or extend remove_by in review"
            )

    for e in errors:
        print(f"toggle-lint: {e}", file=sys.stderr)
    return 1 if errors else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

A small CI lint script that reads a JSON toggle registry, requires owner, creation date and removal date on release toggles, and exits non-zero when one is past its removal date.

When should you avoid feature toggles, and what are the alternatives?

A toggle is worth its cost only when it protects something a smaller tool cannot. Four cases usually fail that test:

  • Small changes that merge in a day. If the change is complete before anyone else touches the code, a toggle only adds a branch you will have to delete later.
  • Branch by abstraction that already isolates the work. If a new implementation sits behind an interface and stays unused until you switch the wiring, adding a runtime flag on top creates a second seam that must be tested and kept in sync with the first.
  • Paths that are not data-compatible. If the new path writes data the old path cannot read, flipping back does not undo the change. Rollback needs a migration plan, not a switch.
  • Permission logic. "Which customers get this feature" belongs to the product's entitlement model, which needs auditing, billing alignment and long-term ownership. If you build it as a toggle, the toggle can never be removed.
Alternative Risk Rollback speed Runtime cost Cleanup burden
Small direct merge Low if the change is small Revert and redeploy None None
Branch by abstraction, no flag Low; the new path is unused until you rewire it Redeploy with the old wiring Indirection only Remove the old implementation
Short-lived feature branch Merge conflicts grow with age Revert the merge None Delete the branch
Canary or blue-green deploy Covers a whole release, not one feature Shift traffic back Extra capacity during rollout None in code
Entitlement system Low once built Change the grant A lookup per request Ongoing, by design

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to trunk based development feature toggles, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.