Feature Flags Across Microservices: Coordinating Cross-Service Rollouts

Why per-service flag evaluation breaks within a single request

A cross-service feature flag is one whose two sides live in different deployables. One service produces the new behavior and another consumes it. A checkout redesign might change both the order API and the payment service. Each service usually embeds its own flag client and evaluates the flag on its own, so a single user request can get two different answers.

Here is how that fails. The order service evaluates new-order-schema, gets true, and writes an order with a split line_items structure. It then calls the payment service, which evaluates the same flag, gets false, and expects the old flat structure. Payment either rejects the request, or worse, reads a default and charges the wrong amount. The order row exists in the new format and the payment record does not match it. The result is a half-applied write that no retry fixes.

Three causes produce this divergence:

• Cache and propagation delay. Flag clients poll or stream updates at their own pace. For seconds or minutes after a change, service A's cache disagrees with service B's.
• Differing evaluation context. Service A targets on user ID and region, while service B only has an account ID. The same rule then buckets the request differently.
• Mid-request updates. A flag can change between the first service's evaluation and the third's, even when every client is in sync.

This matters to any team whose features span more than one service. For them, the requirement is flag consistency: every service must make the same decision for a given request.

How it works: evaluate once at the edge and propagate the decision

The simplest way to get there is to make the decision once. The gateway, or the first service that handles the request, evaluates every flag the request might touch. It then attaches the results to the request as metadata that travels with the trace context:

client → gateway (evaluate, sign) → orders (read) → billing (read) → ledger (read)

Downstream services read the variant from that metadata. They do not ask the flag provider again, so a config change in the middle of a request cannot split its behavior.

What to propagate

Propagate only flag keys and resolved variants, such as checkout-v2=on. A baggage-style entry works well because tracing libraries already forward it.

Leave out the following:

• Rules, percentages and targeting attributes. They leak business logic and user data, and they bloat every hop.
• Unused flags. Headers pass through proxies and load balancers with size limits, so a growing flag list can get requests rejected. Send only the flags the request path needs.

Validate at trust boundaries

At ingress, drop any flag metadata that arrives from outside, then evaluate fresh. Without this step, any client can send checkout-v2=on and opt itself into an unreleased path. Internally, check each key against a known list and each value against its allowed variants. The middleware example below follows this pattern: read, validate, forward.

Trade-offs

• Coupling. The header format becomes a shared contract between services, and changing it requires coordination.
• Dropped metadata. Any hop that fails to forward the entry, such as a queue, a retry library or a hand-built HTTP client, brings the inconsistency back.
• Ingress work. Stripping and validating at every entry point is extra code that must be maintained.
• Coarser targeting. The edge may not know attributes such as account tier, which downstream services hold and would want for finer targeting.

Copyimport type { Request, Response, NextFunction } from "express";

// Internal-service middleware. The gateway deletes any inbound x-flag-decisions
// header from external clients, evaluates fresh, then sets it. This side only reads.
const HEADER = "x-flag-decisions"; // e.g. "checkout-v2=on,billing-retry=fast"
const MAX_LEN = 512; // stay well below proxy and load balancer header limits

// Known flags and their allowed variants. The first variant is the safe default.
const ALLOWED = new Map<string, string[]>([
  ["checkout-v2", ["off", "on"]],
  ["billing-retry", ["control", "fast"]],
]);

export function parseDecisions(raw: string | undefined): Map<string, string> {
  const out = new Map<string, string>();
  if (!raw || raw.length > MAX_LEN) return out; // treat an oversized header as absent
  for (const pair of raw.split(",")) {
    const [key, value] = pair.trim().split("=");
    // Drop unknown keys and unexpected variants instead of trusting them.
    if (ALLOWED.get(key)?.includes(value)) out.set(key, value);
  }
  return out;
}

export function flagDecisions(req: Request, res: Response, next: NextFunction) {
  const decisions = parseDecisions(req.get(HEADER));
  // A missing flag falls back to its default. The service does not re-query the provider.
  res.locals.flag = (key: string) => decisions.get(key) ?? ALLOWED.get(key)?.[0];
  // Rebuild the header from validated entries only, so bad input is never forwarded.
  res.locals.flagHeader = [...decisions].map(([k, v]) => `${k}=${v}`).join(",");
  next();
}

// Every outbound call must forward the header, or the next hop loses the decision.
export function callBilling(res: Response, body: unknown) {
  return fetch("", {
    method: "POST",
    headers: { "content-type": "application/json", [HEADER]: res.locals.flagHeader },
    body: JSON.stringify(body),
  });
}

Express middleware for an internal service that reads the gateway's flag-decision header, discards oversized input, unknown keys and disallowed variants, falls back to safe defaults without re-querying the flag provider, and forwards only the validated decisions downstream.

When propagation is not possible: deterministic bucketing for async and batch work

Some work never sees the originating request. A queue consumer processing an OrderPlaced event, a nightly billing job, or a webhook retry fired hours later has no header to read. Often the message was produced before the flag existed. For these paths, each service has to evaluate the flag itself and still reach the same answer.

Deterministic bucketing makes that possible. Every evaluator hashes the same stable key, such as user, tenant or entity ID, with the flag name, maps the hash to a percentage bucket, and applies identical targeting rules. Given the same inputs, a consumer running at 3 a.m. lands where the API did at noon. The key must exist in the event payload and in every service that evaluates the flag. Random values or session IDs break the scheme: a batch job has no session, and a regenerated session moves the same customer between buckets mid-rollout.

This approach is weaker than propagation. It only produces the same answer if every service has the same inputs, and three things break that in practice:

• SDKs can be configured with different hashing seeds or provider settings.
• A cache that refreshes every few minutes can serve stale rules.
• A rule change halfway through a rollout gives services different answers until every cache updates.

Plan for brief disagreement, and make consumers tolerate both code paths.

Implementing it with OpenFeature as a shared abstraction

OpenFeature is a specification with SDKs in most mainstream languages. It defines one evaluation API: a client that resolves boolean, string, number or object flags against an evaluation context. A provider plugs into that API and connects it to whatever backend actually stores the rules. Go, Java and TypeScript services therefore evaluate checkout.new-pricing with the same call shape. You can replace the backend by swapping the provider registration at startup, and call sites stay unchanged.

The evaluation context carries a targetingKey plus arbitrary attributes. This is where consistency is won or lost. If one service sets targetingKey to the user ID and another sets it to a session ID, a shared provider will bucket them differently. Nothing in OpenFeature will warn you. The standard fixes the interface, not the meaning of what you pass through it. Teams still have to agree on three things:

• Canonical context keys and their source of truth.
• A flag naming scheme, such as domain.feature.
• Which propagation strategy from the previous sections applies to each flag.

Hooks run at defined stages around every evaluation: before, after, on error, and finally. That makes them the right place to enforce these conventions centrally. The example below builds the context from a stable account ID. It registers an after hook that records the flag key, the resolved variant and the reason as span attributes. You can then compare decisions across services in a single trace.

The costs are real. Every evaluation passes through an extra layer of abstraction. Your shared conventions library becomes a dependency that every team must keep current. Getting agreement on context semantics across teams usually takes longer than writing the integration.

Copyimport { OpenFeature, Hook, HookContext, EvaluationDetails, FlagValue, EvaluationContext } from '@openfeature/server-sdk';
import { trace } from '@opentelemetry/api';

// Shared conventions library: every service builds context the same way.
// targetingKey is ALWAYS the account ID, never a session or request ID,
// so a shared provider buckets the same account identically everywhere.
export function buildContext(accountId: string, region: string): EvaluationContext {
  return { targetingKey: accountId, region };
}

// Records each decision on the active span so decisions from different
// services can be compared within one trace.
// Attribute names are a team convention; align them with the OTel version you use.
const recordDecision: Hook = {
  after(hookContext: HookContext, details: EvaluationDetails<FlagValue>) {
    const span = trace.getActiveSpan();
    if (!span) return;
    span.setAttribute('feature_flag.key', hookContext.flagKey);
    span.setAttribute('feature_flag.variant', details.variant ?? String(details.value));
    if (details.reason) span.setAttribute('feature_flag.reason', details.reason);
  },
};

// Registered once at startup. The provider (set elsewhere via
// OpenFeature.setProviderAndWait) is the only part tied to a backend.
OpenFeature.addHooks(recordDecision);

const client = OpenFeature.getClient('checkout-service');

export async function usesNewPricing(accountId: string, region: string): Promise<boolean> {
  // Flag name follows the domain.feature scheme; default is the safe old path.
  return client.getBooleanValue('checkout.new-pricing', false, buildContext(accountId, region));
}

A TypeScript service builds its OpenFeature evaluation context from a stable account ID and registers a global after hook that writes the flag key, resolved variant and reason to the active trace span.

Sequencing a coordinated rollout with expand/contract

The order of deployment decides whether a flag flip is safe. Every service that might receive the new behavior must already understand it before any producer can emit it. The flag should gate behavior only. If turning it on is what makes two services compatible, then partial propagation of the flag value becomes a production incident.

For events and persisted data, tag each message and record with the schema version or variant that produced it. Consumers then branch on the tag instead of guessing from the shape. This tag matters most when the flag goes off again. Records written under the new path still exist, so readers must keep handling them until a migration rewrites or retires them.

Kill switches are not instant. SDKs that poll pick up the change only on their next interval. Streaming connections can lag or reconnect. In-flight requests finish on the old decision, and queued messages were produced under the old value. Turning a flag off also does not undo side effects such as sent emails, charges or schema writes. Plan compensating actions before you ramp.

Ownership, lifecycle and observability of cross-service flags

One owner, one record

Every cross-service flag needs a single owning team, even when five services read it. Shared ownership means nobody removes it. Use a naming convention that encodes the owner and intent, such as payments.checkout-v2-pricing, so a flag found in a stack trace leads to the people who can answer for it. Keep a record next to the flag definition that lists every service that reads it, the expected removal date and the removal plan. Without that list, you cannot know when it is safe to delete the flag.

Removal follows the same ordering as rollout. First, set the flag to its final value everywhere. Then delete the flag checks and the old path from consumers last, after the producers and the edge stop emitting the old decision. If you remove the flag from a consumer first, it falls back to a default that may disagree with what upstream services still send.

Make agreement visible

Each service should record the resolved variant on its spans and logs, not only the edge. For example, add the attributes flag.key=payments.checkout-v2-pricing and flag.variant=on to the active span and to structured log lines. With that in place, you can query a single trace and confirm that every hop saw the same variant.

Alert on two signals:

• Variant disagreement within one trace. This usually means propagation broke or a service is evaluating the flag locally.
• Error rate per variant. An aggregate error rate hides a regression that affects only a small cohort.

Before rollout, exercise the kill switch in a staging environment. Confirm that every listed service picks up the change and how long that takes.

Copyimport logging

from opentelemetry import trace

logger = logging.getLogger("pricing-service")

FLAG_KEY = "payments.checkout-v2-pricing"


def record_flag(flag_key: str, variant: str) -> None:
    # Attach the resolved variant to the active span so a single trace
    # shows what every hop decided, not only the edge.
    span = trace.get_current_span()
    span.set_attribute("flag.key", flag_key)
    span.set_attribute("flag.variant", variant)

    # Mirror the same fields on structured logs so per-variant error rates
    # can be computed from logs as well as traces.
    logger.info(
        "flag resolved",
        extra={"flag.key": flag_key, "flag.variant": variant},
    )


def price_cart(cart, resolve_variant):
    # resolve_variant stands in for your flag client. It should read the
    # decision propagated from upstream rather than re-evaluating locally,
    # so this service agrees with the rest of the trace.
    variant = resolve_variant(FLAG_KEY, default="off")
    record_flag(FLAG_KEY, variant)

    if variant == "on":
        return price_v2(cart)
    return price_v1(cart)  # old path: removed from consumers last

A service records the resolved flag key and variant on the active OpenTelemetry span and on a structured log line, so you can check variant agreement within a trace and compute error rates per variant.

When to use it, when to avoid it, and the impact on cost and engineering time

When it is warranted

Coordinated evaluation pays off when a feature's behavior or data contract spans services. Examples include a new pricing model that the cart, checkout and invoicing services must agree on, a rollout where several teams ship parts of one feature, or a schema change where a reader and a writer must flip together. In these cases a mismatch produces wrong results, not just a different look.

When to avoid it

Use an ordinary local flag if only one service changes behavior. Skip the flag if the change is short-lived and a backward-compatible deploy can ship it safely. If the new behavior is really a new contract, a versioned API or separate endpoint is often cleaner because clients opt in explicitly and the change is visible at the API boundary.

Cost and time

The development cost is real:

• Every hop must forward the decision header.
• Ingress must validate or strip client-supplied values.
• Test combinations multiply.
• Flag debt has to be removed in dependency order by several teams.

Illustrative estimate (assumed numbers, not a benchmark): combinations = 2^n for n independent boolean flags. Assuming 3 concurrent cross-service flags, that is 2³ = 8 paths to cover. With 5 flags it becomes 32.

The savings are also real. Mixed-variant requests stop happening, so you have fewer inconsistent-request incidents to investigate. Variant-tagged traces show which cohort failed, which speeds up diagnosis. Rollout can grow in small, reversible steps across all services at once.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to feature flags across microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.