Learning series · 6 episodes · Staff
The Numbers Were There. The Answers Weren't.
Six connected lessons about telemetry signals, series identities, compression, rollups, percentiles and query overload.
Start series →Follow the decisions in order
- Part 1 of 6
Staff
One refund failed. Why can't the metrics dashboard find that request?Choose which telemetry signal answers fleet health, individual decisions and distributed request paths without exploding metric cardinality.
- Part 2 of 6
Staff
Samples per second stayed flat. Why did the metrics server run out of memory?Diagnose series-creation and label-churn pressure independently of sample volume and design observable safety limits.
- Part 3 of 6
Staff
The server received 160 MB of samples. Why are the compressed chunks so small?Explain why delta-of-delta timestamps and XOR floats compress predictable series and when that advantage disappears.
- Part 4 of 6
Staff
We kept hourly metrics for two years. Why can't we reconstruct last Tuesday's spike?Design retention, rollup dependencies, valid mergeable statistics and safe query precision across raw and coarse intervals.
- Part 5 of 6
Staff
Region A reports 100 ms p99. Region B reports 10 seconds. What's the global p99?Demonstrate the non-mergeability of local quantiles and design correct aggregation of histogram populations.
- Part 6 of 6
Staff
One dashboard opened a 30-day graph. Why did every metrics query slow down?Protect the shared metrics store using read-work cost budgets, fairness, workload isolation and appropriate query resolution.