Scrape jitter

Scrape jitter is the most common cause of oversized TSDB blocks that I have observed. Recall from Chapter 3 how – from the third scrape onwards – timestamp values are stored in the TSDB as they only store the delta of the delta of the sample timestamp. So long as this delta of the delta is 0, the TSDB’s compaction process can save a lot of space by effectively dropping the timestamp value from stored samples that all occur at a consistent delta. With millions of samples, this can add up to gigabytes of storage space in every TSDB block. However, when the delta of the delta is not consistent, this is referred to as scrape jitter.

Scrape jitter is a way to say that scrapes do not occur at consistent intervals. In Prometheus, this often means that they are off by just a few milliseconds. By default, Prometheus will automatically adjust timestamps that are within a 2ms tolerance.

Configuring timestamp adjustments

Whether or not timestamps are...