An hourly average alone throws away too many answers
Suppose the CPU samples for an hour are mostly 30%, but one minute reaches 99%. An hourly average can remain close to 31%. For fleet planning, that's useful. For explaining saturation at 14:02, it is misleading. If you retain the minimum and maximum as well, you can at least say an extreme occurred during the hour, though you cannot reconstruct its exact duration or sequence.
For gauge-like measurements, retain sum, count, min and max for each window when those operations are meaningful. An average over two hours must be (sum1 + sum2) / (count1 + count2), not the mean of two hourly averages when their sample counts differ. Missing samples should change the count and be visible as a coverage problem.
Counters need a different treatment. A cumulative request counter is not meaningfully summarized by averaging its observed values. You need a counter-aware aggregate that accounts for resets and missing intervals. A rollup might retain the increase over a complete window plus reset metadata, depending on the engine. If a reset occurs halfway through an hour, simply subtracting first from last can produce nonsense.
Latency distributions deserve special care. If we keep only hourly p99 values, we cannot merge those into a valid monthly p99. Preserve mergeable histogram buckets or another suitably mergeable distribution sketch at a resolution consistent with the error budget. Otherwise state explicitly that old percentile charts are approximations with weaker semantics.
Retention is a dependency graph
Imagine we implement a raw block store and a rollup worker. A raw block becomes eligible for deletion after seven days. The rollup worker is delayed for two days due to object-store errors. If we delete raw blocks before a verified rollup exists, we have permanently lost the only observations from which that rollup could be calculated.
I would make the lifecycle explicit: raw block becomes complete, rollup is produced under a deterministic time-window identity, validation checks sample coverage and output checksum or manifest, then the retention controller is permitted to retire source data when its policy allows. Failed or incomplete windows remain visible. Do not treat “the job ran” as proof that every interval was summarized.
Use atomic publication of completed rollup blocks or manifests, so queries do not read half-written windows. If corrections arrive late, publish a new version of the affected window and make query routing choose one consistent version. There must be a policy for how far back corrections are permitted. After source expiry, full repair may be impossible.
A specific Thanos trap
Some engines already implement multi-resolution blocks, but their timing rules matter. Thanos's documented compactor produces 5-minute downsampling only when raw blocks have aged roughly 40 hours, and one-hour downsampling once its 5-minute input is old enough, about ten days. Its documentation warns against deleting the source resolution before later downsampling can happen. Thanos Compactor
That means our exact seven-day raw-retention requirement cannot be implemented by blindly setting a seven-day raw limit and assuming the default Thanos one-hour pipeline will always finish. We would need a suitable intermediate resolution retained beyond the one-hour downsampling threshold, or a separate hourly rollup job designed for the required schedule. I would verify the product version and actual compactor inputs before choosing the policy.
Also note that downsampled blocks do not necessarily lower total object-storage footprint while every resolution is retained. Thanos may retain raw, 5-minute and one-hour representations together. Their primary value is making large-range queries cheaper. Storage savings depend on subsequently expiring data at finer resolutions.
Let the query express the precision it needs
A panel covering the last two hours should use raw points if that level is retained. A panel covering a year can use hourly summaries. At the seven-day boundary, avoid mixing raw and rolled-up values in a way that silently changes an expression's meaning. The UI should communicate available resolution and coverage. A missing two-minute incident cannot reappear because someone zooms into an hourly data point.
Alerting should generally read sufficiently fresh, fine-grained data rather than waiting for coarse historical rollups. Capacity planning can tolerate much coarser summaries. The architecture should route each workload to the resolution that answers its question without pretending all queries have identical precision.
My interview answer would define the retention contract per metric type, choose mergeable aggregates, publish complete windows safely, monitor rollup lag, and make retention conditional on data readiness. Before asking “how cheap is two years of storage,” I would ask “which questions must still be answerable after day seven?”
