Why do feature flags evaluate inconsistently across microservices?
A single user request in a microservice system often touches four or five services, and each one may evaluate the same flag independently. Consistent evaluation means every service on that request path reaches the same variant for the same flag. When they disagree, the gateway might render the new checkout while the pricing service still applies the old discount rules, and the result is a broken request that no single service's logs explain.
Three causes account for most divergence:
- Cache skew during a ruleset update. Each SDK instance polls or streams flag configuration on its own schedule. For a few seconds after a change, some services hold the new rules and others hold the old ones, so a request that spans both evaluates against two rulesets.
- Different evaluation context. One service sets the targeting key to the user ID while another uses the session ID. Or a downstream service never receives the
countryorplanattribute that a targeting rule depends on, so it falls through to the default. - Different bucketing implementations. In polyglot stacks, SDKs written in different languages may hash the targeting key differently or use different bucket ranges. The same user then lands at 12% in one service and 61% in another.
Per-request consistency is a different property from per-user consistency over time, which asks whether a user keeps the same variant across sessions. Propagating flag values or context only fixes per-request consistency.
This matters most to backend and platform teams running multi-hop or polyglot paths, where variants drive behavior that must agree across services. For low-risk UI flags evaluated in a single service, brief inconsistency is usually acceptable.
Should you evaluate once at the edge or propagate context?
The naive option, letting each service evaluate the flag independently with whatever attributes it has, is the source of the problem. There are two real alternatives.
Strategy 1: evaluate at edge, propagate values
client -> [edge: eval all flags] --flags: checkout-v2=on--> [svc A] --> [svc B]
Strategy 2: propagate context, evaluate locally
client -> [edge] --ctx: user=123,tier=pro--> [svc A: eval] --> [svc B: eval]
Edge evaluation resolves flags once at the entry service and passes the resolved values downstream. Every hop sees the same variant by construction. The cost is that downstream services cannot target on attributes only they know, such as inventory region or account state loaded mid-request. The edge also has to know about every downstream flag, so shipping a feature in service B means changing the gateway.
Context propagation passes the evaluation context, and each service evaluates locally. Services keep ownership of their flags. Agreement then depends on three things matching across every service: the rules, the hashing and bucketing implementation, and how fresh each service's cached configuration is. Any drift breaks consistency without raising an error.
| Concern | Edge evaluation | Context propagation |
|---|---|---|
| Consistency | Guaranteed | Conditional |
| Service autonomy | Low | High |
| Coupling | Edge ↔ all features | Shared rules/SDKs |
| Header size | Grows with flag count | Grows with context fields |
| Blast radius of bad rule | Edge-wide | Per evaluating service |
| Polyglot risk | Low | High (bucketing parity) |
Use edge evaluation for a small, stable set of cross-cutting flags. Use context propagation when teams own their own features, and invest in parity tests. If a flag only affects one service, skip both approaches.
STRATEGY 1: evaluate once at edge, propagate resolved values
+--------+ +------------------+ flags: +---------+ +---------+
| client |-->| edge | checkout-v2=on | svc A |-->| svc B |
| | | eval every flag |----------------->| use | | use |
+--------+ +--------+---------+ | value | | value |
| +---------+ +---------+
v
+---------------+
| flag rules | one evaluation, values reused downstream
+---------------+
STRATEGY 2: propagate context, each service evaluates locally
+--------+ +------------+ ctx: user=123 +-----------+ ctx +-----------+
| client |-->| edge | tier=pro | svc A |----->| svc B |
| | | no eval |--------------->| eval w/ | | eval w/ |
+--------+ +------------+ | local | | local |
| attrs | | attrs |
+-----+-----+ +-----+-----+
| |
v v
+----------------+ +----------------+
| rules + SDK | | rules + SDK |
| cache copy A | | cache copy B |
+----------------+ +----------------+
parity required: same rules, same bucketing, same cache freshness
Edge evaluation resolves flags once and forwards values, while context propagation forwards attributes and relies on every service matching rules, bucketing, and cache freshness.
How does deterministic bucketing keep a percentage rollout consistent?
A percentage rollout depends on one property: the same subject must land in the same bucket every time, in every service. Deterministic bucketing provides it. The SDK hashes a stable targeting key, such as a user, account or tenant ID, together with the flag key, and maps the result onto a fixed range. If the bucket falls below the rollout threshold, the flag is on.
The sketch below is illustrative only. It does not reproduce any SDK's actual algorithm:
- Concatenate the flag key, a separator and the targeting key.
- Hash the string with a stable, non-cryptographic or cryptographic function that gives identical output on every platform.
- Take the hash modulo 10,000 and compare the result to the threshold. A threshold of 2,500 means 25%.
Using the flag key as salt decorrelates rollouts. Without it, the same 10% of users would receive every experiment.
Hashing a request ID or a random value breaks this. Each call gets a fresh bucket, so a user can see the feature on the gateway and lose it downstream, or flip between page loads. Choosing the key also sets the unit of consistency. If you hash the user ID, users on a shared account can see different behavior. If you hash the tenant ID, the whole tenant moves together.
The subtler risk is polyglot bucketing. A Go service and a Python service can agree on the targeting key and still disagree on the bucket. That happens when their SDKs use different hash functions, string encodings, separators or modulo ranges. Catch this with shared test vectors: a checked-in file of (flag key, targeting key, threshold, expected result) tuples that every service's CI evaluates through its real SDK. Include non-ASCII IDs and boundary values. A parity failure should block the deploy.
# Illustrative only: NOT any SDK's actual bucketing algorithm.
import hashlib
import json
BUCKETS = 10_000 # threshold 2_500 == 25%
def bucket(flag_key: str, targeting_key: str) -> int:
# Flag key as salt decorrelates rollouts across flags.
# Explicit separator and UTF-8 encoding: every language must match these exactly.
material = f"{flag_key}:{targeting_key}".encode("utf-8")
digest = hashlib.sha256(material).digest()
# First 8 bytes as an unsigned big-endian integer, then reduce to the range.
return int.from_bytes(digest[:8], "big") % BUCKETS
def is_enabled(flag_key: str, targeting_key: str, threshold: int) -> bool:
# Hash a stable ID (user/account/tenant), never a request ID or random value.
return bucket(flag_key, targeting_key) < threshold
# Parity check run in every service's CI against one shared, checked-in file.
# Each entry: {"flag": ..., "key": ..., "threshold": ..., "expected": true|false}
# Include non-ASCII IDs and thresholds of 0 and 10000.
def check_vectors(path: str) -> None:
with open(path, encoding="utf-8") as f:
vectors = json.load(f)
failures = [
v for v in vectors
if is_enabled(v["flag"], v["key"], v["threshold"]) != v["expected"]
]
if failures:
# Non-zero exit blocks the deploy.
raise SystemExit(f"bucketing parity failed for {len(failures)} vector(s): {failures[:3]}")
if __name__ == "__main__":
check_vectors("testdata/bucketing_vectors.json")
An illustrative hash-to-bucket function salted with the flag key, plus a CI check that evaluates shared test vectors so every service can prove it buckets identically.
What does OpenFeature standardize, and what does it not?
OpenFeature is a CNCF specification and set of language SDKs that separates how code asks for a flag from who answers. Application code calls a vendor-neutral client to request a typed value (boolean, string, number or object) with a default. A provider behind that client does the actual evaluation against a vendor service, an open-source flag engine or a local file. Swapping providers should not touch call sites, which matters when dozens of services would otherwise import a vendor SDK directly.
The evaluation context is the input to every evaluation. It holds a targetingKey, the stable identifier the provider buckets on, plus arbitrary attributes such as tenant, region or plan. The SDKs merge context set at several levels (global, client and per-call), so a service can attach static attributes once and add request-specific ones at the call site. Hooks run before and after evaluations and on errors, which makes them the standard place for logging, metrics and context validation.
This is where cross-language standardization pays off. Teams publish a shared context-builder convention: the canonical attribute names, which identifier becomes the targetingKey, and how it is normalized (case, encoding). They implement it as a thin helper in each language. A hook can reject or flag evaluations that arrive without a targetingKey.
OpenFeature has limits. It does not put flags or context on the wire between services. It does not guarantee that two services reach the same result. It does not define the bucketing algorithm, which belongs to the provider. Two services using different providers, or different versions of the same provider, can return different answers for the same context. Consistency still depends on your propagation design and on the parity tests described above.
import { OpenFeature, EvaluationContext, Hook, HookContext } from '@openfeature/server-sdk';
// Shared convention: same attribute names and normalization in every service/language.
// The targetingKey is the identifier the provider buckets on.
export function buildFlagContext(req: { userId: string; tenantId: string; region: string }): EvaluationContext {
return {
targetingKey: req.userId.trim().toLowerCase(), // normalize so all services hash the same string
tenant: req.tenantId,
region: req.region,
};
}
// Hook that rejects evaluations missing a targetingKey.
// Throwing in `before` makes the SDK return the caller's default value.
const requireTargetingKey: Hook = {
before(hookContext: HookContext) {
if (!hookContext.context.targetingKey) {
throw new Error(`flag ${hookContext.flagKey}: missing targetingKey`);
}
},
};
// Provider registration (vendor, open-source engine or file) happens once at startup, e.g.:
// await OpenFeature.setProviderAndWait(myProvider);
OpenFeature.addHooks(requireTargetingKey);
const client = OpenFeature.getClient();
export async function useNewCheckout(req: { userId: string; tenantId: string; region: string }): Promise<boolean> {
// Typed call with a safe default; the provider decides the actual value and bucketing.
return client.getBooleanValue('new-checkout', false, buildFlagContext(req));
}
A TypeScript service using the OpenFeature server SDK applies a shared context-builder convention and a global hook that enforces a targetingKey. It then evaluates a boolean flag with a safe default, while bucketing stays the job of whichever provider is registered.
How do you implement flag propagation across HTTP, queues and trust boundaries?
Carry decisions in the same envelope as trace context. W3C Baggage suits this because most tracing middleware already forwards it, for example baggage: ff.checkout_v2=on,ff.ruleset=r1842. A custom header such as x-flag-decisions works too. Keep the payload small, since it rides on every hop and counts against header size limits. Allowlist which flags propagate, because most flags are local to one service and never need to travel.
The ruleset version lets a downstream service detect skew. Set the policy per flag:
- Cosmetic flags: re-evaluate locally.
- Flags that shape the user experience: honor the propagated value so the user sees one coherent result.
- Either case: log a warning with both versions so you can measure how often skew happens.
Queues, sagas, retries and scheduled jobs have no inbound request to read headers from, so put the decision or the evaluation context in the message itself. Then choose between two behaviors. Honoring the enqueue-time decision keeps a multi-step saga coherent. Re-evaluating at execution time lets a kill switch stop work that is already queued, so kill switches usually need this option.
At the public edge, strip every flag header and ff.* baggage key before routing. Otherwise a client can switch features on by sending them. Internal middleware then parses only allowlisted keys. For flags that gate security behavior or billing, sign the decision or re-evaluate it at each zone boundary.
Finally, test every hop. Some proxies, gateways and third-party services drop baggage or unknown headers, so treat a missing value as a normal case.
type HeaderMap = Record<string, string | undefined>; // header names lowercased
const PROPAGATED = new Set(["checkout_v2", "search_rerank"]); // allowlist
const LOCAL_RULESET = "r1842";
// Public edge: drop any flag state the client sent before routing inward.
export function stripAtEdge(h: HeaderMap): void {
delete h["x-flag-decisions"];
if (!h.baggage) return;
const kept = h.baggage.split(",").map((m) => m.trim())
.filter((m) => m && !m.startsWith("ff."));
if (kept.length) h.baggage = kept.join(",");
else delete h.baggage;
}
// Internal hop, e.g. baggage: ff.checkout_v2=on,ff.ruleset=r1842
export function parseFlags(h: HeaderMap): Map<string, string> {
const flags = new Map<string, string>();
let upstream: string | undefined;
for (const member of (h.baggage ?? "").split(",")) {
const kv = member.split(";")[0]; // ignore baggage properties
const eq = kv.indexOf("=");
if (eq < 0) continue;
const key = kv.slice(0, eq).trim();
let value: string;
try { value = decodeURIComponent(kv.slice(eq + 1).trim()); } catch { continue; }
if (key === "ff.ruleset") upstream = value;
else if (key.startsWith("ff.") && PROPAGATED.has(key.slice(3))) {
flags.set(key.slice(3), value); // non-allowlisted keys are ignored
}
}
if (upstream && upstream !== LOCAL_RULESET) {
// Record both versions so skew frequency can be measured.
console.warn("flag ruleset skew", { upstream, local: LOCAL_RULESET });
}
// Empty map is normal (a hop may have dropped baggage): caller then
// re-evaluates locally, or applies its per-flag honor/re-evaluate policy.
return flags;
}
Framework-agnostic TypeScript middleware helpers that strip flag headers and ff. baggage keys at the public edge, and internally parse only allowlisted decisions while logging ruleset-version skew.*
What happens when the flag provider fails or a propagated value is missing?
Every service needs a hard-coded default for each flag it reads, used when the provider is unreachable or the propagated value is absent. The default should be the behavior you are willing to ship to every user at once, which for most new features means "off." OpenFeature's evaluation API requires a default argument on every call for this reason.
Defaults must also match across services. If the checkout service defaults a flag to on and the inventory service defaults it to off, a provider outage leaves the feature half-enabled mid-request: one hop writes the new schema and the next reads the old one. Keep defaults in a shared flag registry and check them in CI, so this mismatch never ships.
The harder decision is how far to trust upstream. Accepting a propagated value keeps evaluation consistent, but a buggy or compromised edge service can turn a feature on everywhere downstream. Re-evaluating locally limits that damage to one service, and the cost is that services may disagree. A practical split is to trust propagated values inside a single trust zone and re-evaluate at zone boundaries.
| Scenario | Recommended behavior |
|---|---|
| Provider down | Use the last cached ruleset if one exists; otherwise use the shared default. Emit a metric. |
| Header missing | Evaluate locally with available context. Log the hop that dropped it. |
| Version skew (unknown flag or variant) | Treat as missing and use the default. Never guess. |
| Malformed value | Reject it, use the default, and alert. Malformed values often point to tampering or a serializer bug. |
How do you observe and test flag consistency, and what does it cost?
Record every evaluation on the span
The failure modes above are hard to see unless each hop records what it decided. OpenFeature hooks run at fixed stages around every evaluation, which makes them the right place to do this. A minimal hook implements only the after stage. It reads the flag key, the resolved variant, and the reason (for example static, targeting match, default, or error) from the evaluation details, and writes them as attributes on the active span. The error stage writes the error code. Use your tracing system's feature-flag semantic conventions for attribute names so that queries keep working when you change vendors. Divergence then becomes a query: find traces where the same flag key has more than one variant across spans. If one span shows reason default while the others show a targeting match, you have probably found a missing or rejected propagated value.
Test the context, then the path
Contract tests feed the same request fixture to each service's context builder and assert that the resulting targeting keys and attributes are identical. Most divergence starts here, for example when one service lowercases an email address and another does not. End-to-end tests register an in-memory test provider with the flag state pinned, send a request through the full call chain, and assert that every span recorded the same variant.
Where it pays
Version checks, signing, hooks and these tests are code that someone has to own. The investment pays off for flags that gate billing, data writes or multi-service workflows. In those cases a mismatch caught in staging, or found in minutes during an incident with a single trace query, is worth the upkeep. For a low-risk UI flag evaluated in one service, it usually isn't. Plain logging is enough there.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to feature flags microservices, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.