Feature Flags in Microservices: Keeping Decisions Consistent

Why feature flags in microservices fail at service boundaries

A distributed feature toggle is a flag whose effect spans more than one service. A new checkout flow might need the cart service to send a field, the pricing service to read it, and the order service to store it. Teams use one flag for the whole change because the feature only works when every participant agrees, and they want to roll it out, target it and roll it back as a single unit.

That agreement breaks when each service evaluates the flag on its own. Each service runs its own SDK instance with its own cache, which can hold a different version of the flag's rules. Caches refresh on independent schedules, so during a rollout change one service may have the new percentage while another still has the old one. Targeting context also drifts: service A may evaluate against a user ID and region, while service B only receives an account ID and hashes a different key into its percentage bucket.

A rollout that splits a request

Service A evaluates new-discount-model as on. It adds a discountTier field to its call to service B. Service B evaluates the same flag in the same request and gets off, because its cache is a few seconds behind or the user hashed into a different bucket. Depending on how strictly B validates input, it either silently ignores discountTier and prices with the old model, or rejects the request as malformed. The user sees a wrong price or an error, and neither service logged anything unusual.

The fix this guide builds on is to make the decision once per request, at the edge or in the first service that needs it, and carry that decision with the request to every downstream hop.

Three coordination patterns and when each fits

That approach is one of four ways to coordinate a flag across services. The right one depends on how far the feature reaches and which service holds the attributes you target on.

Independent evaluation is the default for a flag whose effects stay inside one service. There is nothing to coordinate.

Edge evaluation makes a single decision, so every downstream service agrees. Downstream services lose the ability to target on attributes the edge never sees, such as account tier resolved inside a billing service.

Context propagation passes the targeting key and attributes instead of the result. Each service hashes the key itself. Two services reading different config versions for a few seconds can still disagree, so consistency during propagation windows is only probabilistic.

Centralized evaluation removes divergence in evaluation logic between SDKs. Every request now depends on the evaluator being reachable and fast, so you need defined fallback values.

Copy1) INDEPENDENT -- flag effects stay inside one service
   +-------+     +------------------+     +-------+
   | Svc A |---->| Svc B            |---->| Svc C |
   +-------+     | eval(flag) local |     +-------+
                 +------------------+

2) EDGE EVALUATION -- one decision, result propagated downstream
   +------------+ flag=on +-------+ flag=on +---------+
   | Edge       |-------->| Svc B |-------->| Billing |
   | eval(flag) |         | reads |         | reads   |
   +------------+         +-------+         +---------+
   coupled to edge; Billing cannot target on account tier it resolves

3) CONTEXT PROPAGATION -- pass key + attrs, each service buckets
   +------+ key,attrs +---------------+ key,attrs +---------------+
   | Edge |---------->| Svc B         |---------->| Svc C         |
   +------+           | hash(key)     |           | hash(key)     |
                      | config v7     |           | config v6     |
                      +---------------+           +---------------+
                      versions differ during propagation -> may disagree

4) CENTRALIZED -- sidecar or remote evaluator
   +-------+     +-------+     +-------+
   | Svc A |     | Svc B |     | Svc C |
   +---+---+     +---+---+     +---+---+
       |             |             |     extra hop + per-request dependency
       v             v             v
   +---------------------------------+
   | Evaluator: one path, audit log  |
   +---------------------------------+

Four ways to coordinate a flag across services: evaluate locally, evaluate once at the edge and pass the result downstream, pass the targeting key and attributes so each service buckets deterministically, or have every service call a central evaluator.

How to propagate flag decisions through sync and async hops

The first service that evaluates a flag should pass the result downstream, not ask every hop to recompute it. Keep the decision set small and explicit, such as checkout.new-pricing=on. Put it in a dedicated request header or in W3C baggage next to traceparent. Your tracing instrumentation already forwards baggage, so decisions follow the same path as trace context without extra plumbing in every client.

For queues, copy those decisions into message metadata when you publish. Most brokers support headers or attributes, so payload schemas stay free of rollout concerns.

Inbound flag headers are untrusted. A caller who can set baggage from outside can turn on unreleased features. Strip flag entries at the gateway, or accept only signed values from known internal issuers.

Size forces a trade-off. Propagate every flag and you bloat headers, can hit proxy limits, and expose internal rollout names to anything that logs requests. Propagate too few and downstream services evaluate on their own again, which brings back the divergence you were trying to remove. Forward only the flags whose behavior must agree across more than one service.

Pinning a decision for a whole request or workflow keeps behavior coherent, but a pinned "on" can override a kill switch. Let kill switches take precedence over any pinned value, and check them locally at each hop.

Async consumers have to choose:

• Honor the recorded decision. Choose this when the message shape depends on the flag.
• Re-evaluate at consume time. Choose this when the work is independent and the backlog may be hours old.

Copyfrom opentelemetry import baggage
from opentelemetry.propagate import extract, inject

PREFIX = "flag."
# Forward only flags whose behavior must agree across services.
FORWARDED = ("checkout.new-pricing",)

def edge_context(inbound_headers):
    # Edge: keep trace context, but drop flag entries set by outside callers.
    ctx = extract(inbound_headers)
    for key in list(baggage.get_all(ctx)):
        if key.startswith(PREFIX):
            ctx = baggage.remove_baggage(key, ctx)
    return ctx

def pin_decisions(ctx, evaluate):
    # First evaluator records the decision once for the whole request.
    for name in FORWARDED:
        ctx = baggage.set_baggage(PREFIX + name, "on" if evaluate(name) else "off", ctx)
    return ctx

def is_enabled(name, ctx, evaluate, kill_switches):
    if name in kill_switches:  # checked locally at every hop; beats any pinned "on"
        return False
    pinned = baggage.get_baggage(PREFIX + name, ctx)
    return pinned == "on" if pinned is not None else evaluate(name)

def outbound_headers(ctx):
    # Same carrier works for HTTP requests and queue message headers.
    headers = {}
    inject(headers, context=ctx)  # writes traceparent and baggage
    return headers

def consume(message_headers, evaluate, kill_switches, honor_recorded=True):
    ctx = extract(message_headers)
    if not honor_recorded:  # stale backlog, independent work: decide now
        return evaluate("checkout.new-pricing") and "checkout.new-pricing" not in kill_switches
    # Message shape depends on the flag: use the decision made at publish time.
    return is_enabled("checkout.new-pricing", ctx, evaluate, kill_switches)

Using the OpenTelemetry baggage API to strip inbound flag entries at the edge, pin a small set of decisions, forward them in HTTP or message headers, let local kill switches override pinned values, and choose between the recorded decision and re-evaluation at consume time.

Deterministic evaluation and OpenFeature across polyglot services

When two services evaluate the same percentage rollout independently, they agree only if three inputs match. The first is the bucketing key, such as a user or account ID. The second is the flag configuration, meaning the rules, percentages and salt. The third is the evaluation logic, meaning the hash function and how its output maps to a bucket. Most implementations hash the key with a flag-specific salt and compare the result against the rollout percentage. This is deterministic, but only within a single implementation.

Agreement breaks in quiet ways. A Go SDK and a Python SDK may hash differently or treat a missing attribute differently. An SDK upgrade may change how rules are matched. Even with identical code, one service may receive a config update seconds before another. During that window, the same user is at 10% in one service and 20% in the next.

OpenFeature standardizes the application-facing side of this. Its main concepts are:

• Providers connect the API to a flag backend.
• Evaluation context carries the attributes that targeting rules read.
• Hooks run before and after evaluation, for logging, metrics or validation.
• Transaction context propagation stores request-scoped context once so that later evaluations in the same request pick it up.

A shared context convention matters more than which provider you choose. If one service sends userId and another sends user_id, they will diverge regardless of backend. Define the attribute names once and use them everywhere.

A standard API does not standardize targeting semantics. Write contract tests that evaluate the same flags and contexts in every language and assert identical results.

Copy# Contract test: every service's test suite (Go, Java, TS...) loads the same
# fixture file and must produce the same results. This one is the Python side.
# The provider is assumed to be configured in test setup via api.set_provider(...)
# and to point at the same flag configuration snapshot used by the other languages.
import json

from openfeature import api
from openfeature.evaluation_context import EvaluationContext

# Shared convention: attribute names are defined once and used in every service.
# targeting_key is the bucketing key; never send "user_id" in one service and
# "userId" in another.
CONTEXT_ATTRS = ("accountId", "plan", "region")


def build_context(case: dict) -> EvaluationContext:
    attrs = {name: case[name] for name in CONTEXT_ATTRS if name in case}
    return EvaluationContext(targeting_key=case["userId"], attributes=attrs)


def test_rollout_matches_shared_fixture():
    # Example fixture entry:
    # {"flag": "new-checkout", "userId": "user-1042", "plan": "pro",
    #  "region": "eu-west", "expected": true}
    with open("contract/flag_cases.json") as f:
        cases = json.load(f)

    client = api.get_client()
    for case in cases:
        # The default is the opposite of the expected value, so an evaluation
        # error that silently returns the default fails the test.
        default = not case["expected"]
        result = client.get_boolean_value(case["flag"], default, build_context(case))
        assert result == case["expected"], (
            f"{case['flag']} for {case['userId']}: got {result}, "
            f"expected {case['expected']}"
        )

A Python contract test that builds the evaluation context from a shared attribute convention and checks OpenFeature results against a fixture file that every language's test suite must also pass.

Designing services to tolerate both flag states

Consistent evaluation narrows the window where services disagree, but it cannot close it. Caches refresh at different times, messages sit in queues, and someone will eventually flip flags in the wrong order. The goal is to make that disagreement harmless.

Compatibility in both directions

A producer with the flag on must not break a consumer with it off, and the reverse. In practice, that means three things:

• Make changes additive. Add a new field instead of changing the meaning of an existing one.
• Treat new fields as optional. Consumers should handle their absence and ignore fields they do not recognize.
• Use expand/contract for data changes. Add the new column or field and write both shapes. Migrate readers next, and only then remove the old shape.

If every hop tolerates both versions, the order in which flags flip stops being a correctness issue.

Sequencing

Enable consumers before producers, so that nothing is emitted before something can read it. Disable in the reverse order: producers first, then consumers.

Cleanup needs the same discipline. Remove the flag check from producers first, then from consumers, and delete the flag definition last. That way, no running service evaluates a flag that no longer exists and silently falls back to its default.

The cost

Each shared flag doubles the code paths in every service that checks it. Interacting flags multiply the test matrix, because you need cases for each relevant combination, not just "all on" and "all off." Reviewers also have to reason about states that exist only during rollout. Budget for this cost explicitly, and keep cross-service flags short-lived so the extra paths do not become permanent.

Copy// Order event shared by a producer and a consumer during a flagged rollout.
// Expand phase: the new field is additive and optional; the old field stays.
interface OrderPlaced {
  orderId: string;
  totalCents: number;            // old shape, still written while any reader needs it
  pricing?: {                    // new shape, behind the 'itemized-pricing' flag
    subtotalCents: number;
    taxCents: number;
  };
}

// Producer: with the flag on, write BOTH shapes so consumers in either state work.
function buildEvent(orderId: string, subtotal: number, tax: number, flagOn: boolean): OrderPlaced {
  const event: OrderPlaced = { orderId, totalCents: subtotal + tax };
  if (flagOn) {
    event.pricing = { subtotalCents: subtotal, taxCents: tax };
  }
  return event;
}

// Consumer: handles the new field when present, falls back when absent,
// and ignores any fields it does not recognize instead of rejecting the message.
function readTax(raw: unknown, flagOn: boolean): number {
  const event = raw as OrderPlaced;
  if (flagOn && event.pricing) {
    return event.pricing.taxCents;
  }
  // Flag off, or an older producer sent the message: the old shape is enough.
  return 0; // tax not itemized in the old shape
}

// Rollout: enable the flag in the consumer first, then the producer.
// Disable in reverse: producer first, then consumer.
// Cleanup: drop the flag check in the producer, then the consumer,
// delete the flag definition last, and only then remove totalCents (contract).

A producer writes both the old and new event shapes under a flag while the consumer treats the new field as optional, so either flag state on either side remains safe during rollout and cleanup.

Ownership, observability and failure behavior for shared flags

Every shared flag needs one owning team, recorded with the flag rather than in a wiki. Namespace keys by owner and domain, such as checkout.split-payment-v2, so a key that appears in another service's logs points straight to the team to contact. Readers should register their dependency. A periodic job can compare that registry against flags that have stayed at 100% or 0% for weeks and list removal candidates. Without that list, an unowned flag becomes an implicit contract that nobody dares delete.

For observability, use an OpenFeature after hook in every service to record the flag key, the variant served and the evaluation reason on the active span and in structured logs. Doing it in a hook keeps call sites clean and gives every language the same fields. Within a single trace, you can then query for requests where two services served different variants of the same key. That is the disagreement you need to catch before users report it.

Failure behavior should be decided per flag, not per SDK:

Write the default, the cache validity window and the choice of column into the flag's metadata. On-call engineers then know what each service will do when the flag provider is unreachable.

Impact on development cost and engineering time

Coordinated flags carry a cost that grows with every flag. The upfront work is propagation middleware for each transport you use, naming and context conventions, and contract tests that run each service pair under both flag states. After that, every shared flag adds dual-path code in each service it touches, and it doubles the integration states you must test until someone removes it. That recurring cost is the one teams tend to underestimate.

The comparison you need to make:

Net hours = (incidents avoided × hours per incident) + (debugging hours saved) − (build hours + shared flags × upkeep hours per flag)

Illustrative estimate (assumed numbers, not a benchmark): Assume 120 hours to build propagation and contract tests, 6 hours of upkeep per shared flag, and 10 shared flags a year. Year one costs 120 + 60 = 180 hours.

Now assume this avoids 3 mixed-version incidents at 30 engineer-hours each (90 hours). Assume flag-aware traces also save 4 debugging hours per flag (40 hours). The return is 130 hours, so year one is about 50 hours short of break-even. In year two the build cost is already paid, so 60 hours of upkeep buys 130 hours back.

Cheaper alternatives often fit better:

• Contained single-service flags: use these when only one service changes behavior and its contract stays the same.
• Versioned APIs: use these when the change alters a contract. Consumers then opt in on their own schedule.
• Coordinated deploys: use these for rare changes when one team owns both services.

The coordination effort is not worth it in three cases: few flags cross service boundaries, flags live only for days, or one team ships all the affected services together.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to feature flags microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.