Federating Prometheus

The elusive “single pane of glass.” Everybody wants it. Every software vendor purports to sell it. The dream is to have one place you can go to see all of your monitoring data in just that single location. Sharding Prometheus may seem antithetical to that but through federation, we can still achieve it.

What is federation? Federation is the process of joining together metrics from multiple sources into a central location. It is useful for aggregating your metrics into a centralized Prometheus instance. They may be a 1:1 match of the metrics present in the “lower” Prometheus instances, but they can also have PromQL query functions applied to them to perform series aggregation and consolidation before storage in the “higher” Prometheus instance(s).

Should I federate?

If you’re federating because you’ve sharded your Prometheus instances, you’ve probably already run into the issue where one Prometheus...