One refund failed. Why can't the metrics dashboard find that request?

There are two kinds of question hiding inside this request

First, the operational question: are refunds slower, which region fails, and did the release change errors? You can answer those by counting requests and summarizing durations across well-chosen dimensions. You rarely need to remember each individual HTTP request to calculate that picture.

The second is investigative: what happened to refund R-4826, which policy version evaluated it, and which downstream call failed? This question needs a particular execution history. A time series labelled only by route=/refund, status=409 and region=west cannot reconstruct that history. It was never designed to.

Metrics and events are not two databases containing the same information in different formats. They are different representations with different information loss. Aggregation makes metrics cheap to query at scale precisely because it discards much of the identity of individual operations.

Imagine that one latency metric uses 60 route templates, 5 status classes and 4 regions. Those dimensions permit 1,200 combinations, although only combinations actually observed become series. If somebody adds 500,000 distinct customer IDs, the potential combinations balloon to 600 million. They won't all necessarily appear, but it is now possible for traffic to create an enormous number of distinct series without increasing the request rate.

The important cost in a typical labelled metrics store is not simply the number of samples. Each unique identity needs associated index and active-series state. OpenTelemetry's metrics model also maintains aggregation state per attribute combination. The problem appears before long-term disk retention becomes interesting. OpenTelemetry metrics and Prometheus label guidance explain the underlying choices.

I would keep the metrics deliberately boring

For this service, I would instrument a request counter, an in-flight request gauge if needed, and a duration histogram. The labels might be route template, outcome class, region, and a bounded error category. They must be finite, useful dimensions that engineers actually group or filter by. A URL such as /refund/R-4826 should be normalized to /refund/{refundId} before it becomes a route label.

The metric answers whether the system is healthy. A count grouped by error category can show that POLICY_REJECTED increased after deployment. A latency histogram supports questions about tail behaviour that an average duration cannot answer. Histograms have their own cardinality and resolution costs, so I would not create dozens of arbitrary label dimensions for them either.

Now create a structured event for the individual decision. A useful event might contain the refund ID, an event timestamp, the decision outcome, the policy version, a correlation ID, and a restricted diagnostic reason. Keep sensitive customer data out of the payload unless retention, access control and privacy policy justify it. The event store needs a way to retrieve the specific refund history, perhaps an index by refund ID and time. It may be a log/search store or an operational event table, not necessarily a metrics TSDB.

The event pipeline has different costs. At 40 million events per day and an assumed 500 bytes of logical payload per event, that's about 20 GB per day before indexes, replication, compression and storage overhead. We should size those separately, rather than pretending events are free because they did not multiply metric labels.

Where traces help, and where they don't

Distributed traces answer a third variation of the question: which services and spans did this request pass through, and where was time spent? A trace ID can connect the refund event to the authorization service, fraud policy evaluator and ledger call. A trace may be sampled, however. Head sampling can omit rare failures. Tail sampling can retain interesting traces after seeing their outcome, but it introduces buffering, routing and sampling-policy complexity. Sampling is not a substitute for an authoritative audit record.

If compliance requires proving why a refund was denied, I would persist the decision event independently of tracing and define its delivery and retention guarantees. If the team only needs representative latency breakdowns, sampling traces is reasonable. If product needs a real-time count of all denied refunds, use a metric or a properly maintained aggregation, not a count of sampled traces.

This is also a useful place to separate observability from product data. A customer-facing refund history has its own correctness contract and access controls. Do not make the metrics backend the only source of a business decision that somebody may challenge later.

What I would ask before choosing a store

I would ask support to show three actual investigative queries. Is it refundId = x? Is it all failures for a policy version over a day? Is it a free-text search through rejection details? Each favors different indexes and retention choices. I would also ask the security team who may retrieve events and for how long. Search does not justify indefinite retention.

Finally, connect the signals carefully. The metric should point the engineer toward the failure category. An exemplar or correlated event may lead to one trace. The event reveals the individual decision. The authoritative payments database determines the current refund state. Each tool answers what it knows.

For the interview, my design is not “use Prometheus plus Elasticsearch” by default. It is: use bounded, pre-aggregated metrics for population questions, searchable events for individual history, and traces for execution paths. Choose concrete systems only after the query, volume, privacy, and durability contracts are clear.