Prometheus’ limitations

When you first start using it, Prometheus may seem like it can do anything. It’s a hammer and everything looks like a nail. After a while, though, reality sets in and the cracks start to show. Queries start to get slower. Memory usage begins to creep up. You’re waiting longer and longer for WAL replays to finish when Prometheus starts up. Where did I go wrong? What do I need to do?

Rest assured, you did nothing wrong and you’re not alone. Prometheus, like any other technology, has limitations. Some of them are specific to Prometheus and others are limitations of time series databases in general. So, what are they? Well, the two we need to care about the most are cardinality and long-term storage.

Cardinality

Cardinality refers to the measure of unique values of a dataset. High cardinality indicates that there is a large number of unique values, whereas low cardinality is a small number of unique values.

Examples of high...