Every dashboard is green. Where did 12,000 account updates go?

Interview question. Your account service shows normal CPU, healthy p99, low error rates, and a queue with no unusual backlog. Support reports that thousands of profile changes never appear in customers' search results. Investigate correctness, freshness, completeness, and silent drops rather than assuming good infrastructure metrics mean the system is healthy.

A green dashboard is a report about the things somebody decided to measure. It isn't a certificate of correct behavior. Imagine a customer changes their shipping address and receives a success response. The authoritative customer row is updated. A downstream search index still contains the old address. The update pipeline accepted every message it processed, so its error count is zero. From the perspective of the customer, the feature is broken.

The first thing I'd establish is which system owns the truth. Suppose a relational database is authoritative for profiles, a change-data-capture stream carries updates, and search is a derived view. Search being stale for a few seconds may be expected. Search remaining stale for six hours is not. We need an explicit freshness contract: perhaps 99% of successful profile changes should become visible in search within sixty seconds. The exact percentage and deadline are business decisions. Without that contract, "the system is healthy" doesn't have a customer-facing meaning.

Now trace one reported profile using a secure internal identifier. Can we prove the authoritative database committed the change? Did CDC observe the corresponding log position? Was the record published to the expected topic? Did the consumer advance past it? Did the index store the newer version? Does the search endpoint retrieve the index we just updated, or a stale replica? These are independent boundaries. Finding the earliest boundary that disagrees with the previous one is more useful than seeing that all processes are running.

Suppose the update stream contains 500,000 events per hour. A new schema validator mistakenly filters events whose address field is null, treating them as irrelevant. During one deployment it drops 12,000 valid updates and still commits their offsets. Consumers keep up, the queue drains, and broker depth never increases. There is no processing exception because the code chose to discard them. Normal latency and low error rates are completely consistent with missing outcomes. An alert on queue depth alone cannot detect something that is already gone from the queue.

The detection mechanism must compare the expected work with the observed outcomes. Count successful authoritative commits by bounded operation type, then count change-feed publications, consumer decisions, accepted index writes and visible document versions. These counts won't always match instantaneously because of retries, batching, deletes and legitimate filtering. We should compare matched windows and compatible cohorts, with a delay allowance, and define explicit terminal reasons for every intentionally skipped item. A skipped event is a business outcome requiring a reason, not an invisible optimization.

A reconciliation process gives stronger evidence. Periodically sample changed profile IDs from the authoritative database and compare immutable version numbers or hashes with the derived index. Escalate to a bounded complete audit of a partition or time range when the sample finds drift. The comparison must avoid mistaking a legitimate concurrent update for corruption. Pin a database snapshot or compare version ordering and revisit records that were changing during the check. A random sampling process that finds no mismatch can reduce uncertainty, but cannot prove zero missing records.

Be careful with versions. If an older CDC event arrives after a newer one, the search writer should reject the regression using an appropriate source version or sequence. If the original bug caused an event to be dropped, replay from a retained authoritative stream or reindex from the database. The recovery job must not overwrite more recent versions with older snapshots. This is where observability becomes operational design: unless you retain a trustworthy source and a safe repair protocol, detecting a mismatch may leave you without a way to fix it.

What should we alert on? I'd build a small set of customer-outcome signals: age of the oldest unreflected successful update, fraction of updates reaching visibility inside the deadline, unmatched committed source changes after grace time, and sampled cross-store version disagreements. These should be measured by region or pipeline where that distinction affects mitigation. Avoid a time-series label per profile, because high-cardinality identifiers belong in secured investigative events, not metric label sets.

At Staff level, I want the failure-domain question too. If the CDC connector is wedged, throughput falls and freshness age rises. If a code path silently skips updates while acknowledging offsets, consumer lag and queue depth can look perfect. If the index applies changes but queries hit an old alias, ingestion telemetry looks good yet users still see stale results. A single "indexing successful" counter cannot distinguish these cases. Instrument the boundaries the user experience actually crosses.

I would ship a targeted repair that preserves monotonic document versions and audit the affected time window. Then I'd add an acceptance test that inserts a valid profile update with the previously problematic field and waits for the search version, rather than merely asserting a successful HTTP response. The goal isn't to make the dashboards red. It's to ensure a green dashboard represents the contract customers thought they were getting.