The feature flag says off. Why are some customers still seeing the feature?

Interview question. A company uses feature flags in 1,000 services for gradual releases and emergency kill switches. A flag is disabled globally at 14:00, yet some users see the new checkout path for another ten minutes. Design local evaluation, consistent rollout assignment, update propagation, safe fallback, and evidence that a kill switch took effect.

A product manager looks at the control plane and sees new_checkout=false. The incident channel, however, contains screenshots of the new flow. One engineer concludes the flag system is inconsistent. Another proposes calling the flag server on every request so the answer is always current. Both reactions skip an important question: what did the flag control plane promise about when a changed value becomes active in each service, browser session, or request?

Feature flags are configuration with a decision function attached. A Boolean key alone is easy to store. The harder part is evaluating rules over user, tenant, region, application version and rollout percentage while keeping the decision stable enough for a customer journey. A request must be able to explain which ruleset was evaluated, which context fields were used, and which variant was returned. Otherwise a central dashboard and a user's observed behavior cannot be compared meaningfully.

A low-latency implementation usually evaluates flags locally in the service or application using a cached, versioned ruleset. A streaming subscriber or polling agent updates that ruleset asynchronously. The request supplies an evaluation context, such as tenant identity, customer identity or country, and receives a typed value plus useful evaluation metadata. OpenFeature defines a provider-neutral evaluation API and an evaluation context with a targeting key. It does not, by itself, provide a centralized flag distribution service or guarantee one propagation time across implementations. Keep the contract of your chosen provider explicit.

Now consider a 10% rollout. A tempting implementation is random() < 0.1 on every request. That means a user may see the new checkout for one click and the old checkout for the next. Instead compute a stable bucket from a canonical targeting key, flag identifier and rollout salt, then compare the bucket with the allocated range. For example, hash (flagId, tenantId) when an entire tenant must stay together, or (flagId, userId) for user-level assignment. The exact hash and normalization must be consistent across SDK languages. Changing salts or targeting-key semantics can reassign everyone, even if the percentage stays at 10%.

Sticky rollout assignment is not the same as cross-service consistency. Suppose the gateway has ruleset 41 and the payment service has ruleset 42. The gateway evaluates checkout variant A, but payment independently evaluates variant B. If those variants have incompatible request formats, one request can fail even though both evaluations are internally correct. For cross-service protocols, prefer backward-compatible behavior independent of flag state. Where the same decision must be consistent for one transaction, evaluate once at a trusted boundary and propagate the decision and experiment version, or enforce a compatible operation contract between components. Do not treat a client-provided flag as an authorization grant.

Why was the flag still on ten minutes after disablement? Trace the version timeline. At 14:00 revision 43 is approved. Some SDKs receive the stream update immediately, others have a failed subscription and poll every fifteen minutes. A mobile client has a cached assignment for the session. An application worker may have swapped its ruleset but an in-flight request captured revision 42 at request start. These are distinct causes. Record the active revision with evaluation results and expose age of the locally applied ruleset. Count the fraction of healthy instances using the new revision, not merely successful pushes to the distributor.

An emergency kill switch needs a tighter contract than an ordinary visual experiment. If a new recommendation panel is broken, a few minutes of stale enablement may be acceptable. If the flag controls whether a payment may be authorized, a stale true might violate a security or financial constraint. Use an independent authoritative safety control for critical approvals, or define a short-validity policy that fails closed once its freshness deadline passes. Calling the flag system itself a kill switch does not make propagation instantaneous. Prove the end-to-end bound, including edge caches, SDK reconnects and process-level adoption.

Missing evaluation data creates another failure. Suppose 20% rollout is defined per tenant, but a downstream background job lacks tenantId. Silently hashing an empty string can place all such jobs in one bucket. Using a random UUID on each call makes results unstable. Define an explicit fallback value, log a bounded reason code, and prevent missing-context paths from unintentionally enabling sensitive behavior. Avoid logging raw user attributes in evaluation telemetry. The right targeting key can often be a stable pseudonymous identifier rather than personal data.

Now design a rollout timeline. Publish the rule as immutable revision 42, check schema and targeting syntax, evaluate it in a shadow mode against sampled real contexts, then activate it for a small cohort. Observe technical and business outcomes, including checkout completion, user-visible errors, and variant exposure counts. Increase only after adequate evidence. When a change is paused, do not automatically sweep old variant state out of long-running jobs that already committed to that behavior. Decide whether active sessions should finish on the original variant and when a hard stop overrides them.

Finally run a kill-switch drill. Turn off the flag, partition 10% of SDK subscribers, crash one local cache, and generate a mixed-version checkout request. Verify the stated propagation bound or permitted degraded behavior, detect stale revisions, and prevent incompatible evaluations from corrupting payment state. A Staff-level answer distinguishes assignment, evaluation, delivery and authority. The dashboard showing off is only the beginning of the state transition, not proof that customers have stopped seeing the feature.