A critical component of the Prometheus server is its time series database. Being able to analyze the usage of this database is essential to detect series churn and cardinality problems. Churn, in this context, refers to time series that become stale (for example, from the origin target stop being collected or the series disappearing from one scrape to the next), and a new series with slightly different identity starts being collected next. A usual example of churn is related to Kubernetes application deploys, where the pod instance IP address changes making the previous time series obsolete, and replacing it with a new one. This impacts performance when querying, as samples with – possibly – no relevance are returned.
Thankfully, there's an obscure tool within the source code for the Prometheus database that allows analyzing...
