Skip to content
ArchCrux

Capacity, Isolation & Operations · Staff

Region A reports 100 ms p99. Region B reports 10 seconds. What's the global p99?

The Numbers Were There. The Answers Weren't. · Part 5 of 6 · 6 min read

Read the worked explanation ↓

Interview question. A payment API serves three regions. Each region publishes its p99 latency, and the global dashboard takes the arithmetic average of those three numbers. Engineers use that line to decide whether the service is meeting its latency SLO. Explain whether it is valid. Then design a metrics representation that can compute a meaningful global percentile as traffic shifts between regions.

Before discussing histogram implementations, I would write a tiny request distribution on the board. It is easier to expose the error with actual observations than with the phrase “percentiles are not additive.”

For simplicity, consider just two regions. Region A serves 1,000 requests, each taking 100 milliseconds. Region B serves 10 requests, each taking 10 seconds. A's p99 is 100 milliseconds. B's p99 is 10 seconds. Their arithmetic average is 5.05 seconds. Is that anywhere near the global p99?

Combine the actual 1,010 requests and sort them. The first 1,000 are 100 milliseconds. The last 10 are 10 seconds. With the nearest-rank definition, the 99th-percentile observation sits at rank ceil(0.99 × 1010) = 1000. The global p99 is 100 milliseconds, not 5.05 seconds.

Two regions' p99 values conceal their request volumes, while merging bucket populations yields a different global tail estimate.
Two regions' p99 values conceal their request volumes, while merging bucket populations yields a different global tail estimate.

The example isn't an edge case about floating-point precision. The dashboard discarded the distribution and the traffic weights. The local quantiles are too small a summary to reconstruct the population's quantile.

Think first

Three regions have different request volumes and different latency distributions. Design a global five-minute p99 dashboard and a 300 ms SLO burn-rate alert. Explain instrumentation, aggregation, bucket alignment, partial-region failure and the costs of label dimensions.

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

The weighted average is also wrong

A reasonable interviewer follow-up is “What if we weight each regional p99 by request count?” In this example, that produces approximately 198 milliseconds, which is still not the global p99. Weighting fixes one obvious imbalance but not the fundamental information loss. Knowing one percentile from each region doesn't tell us how many observations sit in every range of latency.

For a more general illustration, two regions can report the same p99 while having quite different shapes immediately below that threshold. Their combined p99 can therefore change when the traffic weights change, even when neither regional p99 moves. This is why I wouldn't build a global latency SLA out of an average of locally calculated quantiles.

The correct design starts with a mergeable representation of the observations, not the final regional answer.

Merge populations first, calculate the quantile second

A classic histogram records cumulative observations under configured latency boundaries. Suppose buckets end at 100 ms, 300 ms, 1 s, and +Inf. Each region reports a cumulative count for each boundary. To combine regions, sum matching buckets, then estimate the quantile from the combined distribution.

For Prometheus-style classic histograms, the shape is conceptually:

histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

The bucket counts are counters, so the query derives rates over a consistent time window and sums by the le boundary. If we need a global result across all regions, we intentionally remove the region dimension during the aggregation. If we want p99 per API route, retain the route label along with le.

Prometheus also supports native histograms, where a histogram observation is stored as a structured sample rather than a separate time series for each classic bucket. The global form uses compatible native histogram streams summed before the quantile calculation. Instrumentation support, bucket resolution and compatibility still need verification. Prometheus histograms guidance.

Be precise about what a histogram proves

A histogram usually does not store every exact latency. A quantile interpolated from buckets is an estimate whose accuracy depends on the bucket boundaries or schema resolution. If the business cares whether 99% of requests complete within 300 milliseconds, there is an even more direct calculation: count observations at or under the SLO threshold, divided by all observations over the same interval. That threshold fraction avoids pretending we know an exact p99 when a histogram's resolution is coarse.

Buckets must agree across the populations we merge. If one region uses 100, 300 and 1,000 ms boundaries while another uses 125, 500 and 2,000 ms, blindly adding the same named bucket labels can be incorrect. A native histogram implementation may be able to reconcile compatible schemas, but the details are product-specific. Define an instrumentation contract rather than letting every service improvise latency buckets.

Time alignment matters as much as bucket alignment. A five-minute window in one region and a ten-minute window in another do not describe the same population. Exported summaries computed over application-local sliding windows may also use different decay semantics. A global panel should describe the exact interval and which request classes are included.

Cost and cardinality have not disappeared

Classic histograms multiply the number of time series by their bucket count, plus the count and sum series. If every histogram is labelled with raw user IDs, no quantile function will rescue the cardinality explosion. Use bounded dimensions and select bucket layouts around meaningful thresholds. Native histograms can reduce some series-count overhead, but their samples have their own size and processing costs.

I would also ask how partial failure should display. If the APAC metrics collector is down, a dashboard computed over just the remaining regions should not be labelled “global” without a coverage warning. A low global p99 because the slowest region stopped reporting is a measurement failure, not a performance improvement.

In a Staff interview, this is an opportunity to distinguish the mathematics from the monitoring tool. Averaging percentiles is invalid regardless of whether the dashboard is Prometheus, Datadog or an in-house engine. The design should preserve mergeable counts or distributions, combine matching populations, compute the quantile afterward, and communicate estimation error and missing coverage.

Go deeper

A quantile is an order statistic. A single local quantile does not preserve enough information to calculate a quantile of the union. Even weighted means of local p99 values are not generally meaningful as global p99 estimates. Histogram bucket populations are mergeable when intervals and bucket semantics align. Classic Prometheus buckets are cumulative counters and require a consistent label set and counter-aware rate calculation. Native histograms encode distributions differently and support aggregation when schemas are compatible.

For tight SLO decisions, prefer a directly measured fraction at the threshold when possible. For display, annotate approximate quantiles with their resolution. Test uneven traffic, missing regions, counter resets and bucket configuration drift. Do not reinterpret a sampled set of traces as the full request population unless the sampling design justifies that inference.

Check your reasoning

  • Show that 1000x100ms plus 10x10s yields rank-1000 p99 of 100ms.
  • Reject both unweighted and request-weighted averaging of local p99s.
  • Merge compatible histogram bucket counts over one aligned time window before quantile estimation.
  • Explain histogram bucket error and direct SLO threshold fraction.
  • Describe missing-region data, bucket drift, cardinality and instrument support.

Common mistake: Traffic weighting converts regional p99 estimates into the exact global p99.

Which aggregation can produce a meaningful global latency percentile from multiple regions?

Read the answer explanation

Combine matching histogram populations first, then estimate the quantile — A percentile is an order statistic. Local quantiles lose the distribution information needed to recover the union quantile.

Apply it elsewhere

A fleet of inference servers publishes p95 token latency and request throughput per GPU pool. Traffic is concentrated on two pools. Design a meaningful fleet-wide latency indicator and describe what happens when histogram bucket schemas differ.

Compare your approach

Compute the 1,010-request example correctly, disprove arithmetic and weighted averaging, distinguish histogram aggregation from quantile aggregation, give correct PromQL ordering, discuss bucket precision and time windows, and make missing-region coverage explicit.

One more constraint

Does request-count weighting make the average of regional p99 values equal to the global p99?

Quick revision

Percentiles are not additive. If 1,000 requests take 100 ms and 10 requests take 10 seconds, averaging regional p99s gives 5.05 seconds, but the combined nearest-rank p99 is 100 ms. Weighting regional p99s by traffic is still wrong. Export compatible histograms, combine buckets or native histogram populations over the same time range, then calculate the quantile. For a fixed 300 ms SLO, a proportion below the threshold can be more meaningful than an interpolated p99. The follow-up is what a “global” dashboard should show when one region stops reporting entirely.

References