Skip to content
ArchCrux

Capacity, Isolation & Operations · Staff

The server received 160 MB of samples. Why are the compressed chunks so small?

The Numbers Were There. The Answers Weren't. · Part 3 of 6 · 6 min read

Read the worked explanation ↓

Interview question. A monitoring team stores timestamps and floating-point values for thousands of sensors. Each uncompressed observation can be represented by an eight-byte timestamp and an eight-byte value. Their architect claims a time-series engine can use much less than 16 bytes per sample without discarding observations. Explain delta-of-delta timestamps and XOR float encoding. Where do these techniques stop helping?

I would start with four timestamp gaps, not with a compression benchmark. Suppose a sensor records at 0, 15, 30, 45 and 61 seconds. The gaps are 15, 15, 15 and 16. Store the first timestamp, then a first gap, then the changes between gaps. Those changes are 0, 0 and +1.

That last sequence has a useful property. A regularly sampled sensor tends to produce lots of zero changes. Even if the device is slightly late once, the change is often small. An encoding can give the common cases short bit patterns and reserve longer representations for larger changes.

Regular sample gaps become zero delta-of-delta values, while unchanged floating-point bit patterns produce zero XOR.
Regular sample gaps become zero delta-of-delta values, while unchanged floating-point bit patterns produce zero XOR.

This is not arithmetic that magically makes all timestamps smaller. It is a representation tuned to a statistical pattern. If the sampling schedule has no regularity, the delta-of-delta values can be large and the compressed data grows. That is one reason compression ratios depend on workloads rather than on the word “time-series.”

Think first

A fleet has 200,000 temperature sensors. Half report every 10 seconds with slowly changing readings. The rest report irregularly and produce nearly random 64-bit floats. Design a benchmark that compares raw pairs, delta-of-delta and XOR compression. What numbers would convince you to adopt the codec?

Take a few minutes to form an approach, then compare it with the worked explanation.

Reveal the worked explanation

The floating-point trick is more surprising

A naive design might subtract the previous floating-point value from the next one, then store the numerical difference. But approximate numeric similarity is not always the same as similarity of IEEE 754 bit patterns, and subtraction introduces its own representation concerns.

The Gorilla-style approach instead compares the 64-bit encodings of consecutive floating-point values using XOR. If two values are bit-for-bit equal, the XOR result is zero. That can be represented very cheaply. If only a few bits changed, the XOR may contain long runs of leading or trailing zeros. The encoder can describe a significant-bit window and store only the bits needed for the change.

Take the readings 25.0, 25.0, 25.0, 25.5. The first value establishes a full representation. The two repeated readings have zero XOR. The final change alters the bit pattern, so it requires additional information, but still may be cheaper than another full 64-bit value. If values jump unpredictably or contain noisy low-order bits, the advantage becomes smaller.

The important word is lossless. XOR encoding lets the decoder reconstruct the exact stored floating-point bit patterns under that encoding's supported semantics. It is not rounding sensor readings or estimating missing observations. This is different from downsampling, which deliberately reduces time resolution and cannot usually recover every original sample.

The original Gorilla paper describes delta-of-delta timestamps and XOR floating-point values as key ingredients in a monitoring-oriented TSDB. Prometheus's published chunk format also documents these techniques in its XOR chunks. These are concrete implementations, not a guarantee that all TSDBs use an identical bitstream. Gorilla, VLDB 2015 and Prometheus chunk format.

Compute the upper bound before claiming a win

Ten million observations represented as a raw pair of two eight-byte values require 160 million bytes of sample payload, which is about 153 MiB. That calculation ignores metric labels, series metadata, indexes, chunk headers, storage replication and WAL overhead. It is a baseline for the two fields, not a prediction of total TSDB disk usage.

Suppose most timestamps follow a 15-second interval and most sensor values either repeat or change slightly. Many later observations can fit into far fewer than 128 bits. In a different workload, each reading may use a new random timestamp interval and a noisy float. The same algorithm can then spend much more space per observation.

Also notice that compressing a series well does not make a million one-sample series cheap. Each isolated series still needs identifying metadata, and it has little history from which to learn repeated timestamp gaps or value patterns. That connects to our previous episode about cardinality, but the core mechanism here is per-series compression.

Walk one sample through a real storage path

In a Prometheus-style design, a freshly ingested sample reaches an in-memory head structure and a write-ahead log for crash recovery. The compressed chunk representation and the on-disk block index are not simply one big ZIP file. Recent WAL bytes can be much larger than compacted chunks. Block compaction may rearrange chunks and metadata later.

A query by metric labels first has to find the relevant series. The index tells the engine which series match. The engine then locates chunks for the time range and decodes their sample sequence. Compression helps with IO and memory bandwidth, but the index and the number of matching series can still dominate a query. This is why a TSDB needs both efficient data encoding and efficient access paths.

There is also an ingestion-time complication: out-of-order timestamps. A decoder optimized for monotonic timestamps and nearby samples does not necessarily accept arbitrary late samples into an existing compressed sequence. Different engines buffer, reject, rewrite or use separate backfill mechanisms. It is unsafe to infer supported late-arrival behavior from the compression algorithm alone.

What I would optimize next

If the business needs an exact audit of every sensor reading, preserve raw points for the required period. If it needs only monthly trends, retention and rollups may save more query work than another encoding tweak. Those are separate decisions with separate correctness costs.

I would benchmark compression using realistic sensor distributions, timestamp jitter, counter resets, special floating-point values and series churn. Measure end-to-end ingestion CPU, query decode cost, index and WAL sizes, not only bytes per compressed sample. A Staff-level answer should be able to explain why the bits are small and why that does not automatically make the entire storage service cheap.

Go deeper

In Prometheus's classic XOR chunk format, the first timestamp and float establish a starting state. The second timestamp uses a delta. Later timestamps encode the delta of consecutive deltas. Subsequent floats encode the XOR of their current and previous 64-bit representations. Control bits select zero, reused significant-bit windows, or a new window. An implementation must handle special floating-point values and chunk boundaries with its actual format rules rather than inventing a generic codec.

Compression ratio improves for regular sampling and repeated values but can deteriorate for timestamp jitter, high-entropy floats, and short-lived series. This is lossless sample encoding, not time-window aggregation. An on-disk TSDB may also use distinct encodings for native histograms, exemplars and other sample types.

Check your reasoning

  • Compute example timestamp gaps and second differences.
  • Explain why repeated float bit patterns create a zero XOR rather than decimal subtraction.
  • Distinguish lossless encoding from historical downsampling.
  • Account for series metadata, WAL, indexes and compression limits.
  • Test jitter, random floats, short-lived series and out-of-order data.

Common mistake: XOR compression preserves fewer decimal digits to achieve its space savings.

How can a time-series codec losslessly encode regular timestamps and repeated floating-point readings compactly?

Read the answer explanation

Encode predictable time gaps and bit-pattern changes with variable-length control fields — Delta-of-delta and XOR coding exploit repeated patterns to encode the original samples compactly, with workload-dependent ratios.

Apply it elsewhere

A stock exchange wants lossless price and timestamp storage but experiences bursts, corrections and irregular trading gaps. Explain where the same codec may work, where it may degrade and whether late corrections require a distinct ingestion path.

Compare your approach

Reconstruct the timestamp delta-of-delta example, explain exact-bit XOR decoding, separate payload from metadata/WAL, distinguish lossless compression from rollups, and discuss irregular timestamps, noisy floats, late arrivals and CPU tradeoffs.

One more constraint

Does XOR float compression work by subtracting the previous decimal value and rounding the difference?

Quick revision

With regular sampling, timestamp gaps are mostly constant, so the change in the gap is often zero. Delta-of-delta encodes that cheaply. XOR encoding compares consecutive IEEE 754 float bit patterns. Identical values yield zero, while similar bit patterns may have compact significant-bit windows. Both techniques can preserve the exact sample values rather than downsampling them. Compression suffers with jitter, random values and short-lived series. For a complete capacity estimate I include the WAL, labels, indexes, replication, and decode cost, not just the compressed chunks. The follow-up is how late samples interact with a monotonic compressed sequence.

References