Having a global view using federation

When you have multiple Prometheus servers, it can become quite cumbersome to know which one to query for a certain metric. Another problem that quickly comes up is how to aggregate data from multiple instances, possibly in multiple datacenters. Here's where federation comes into the picture. Federation allows you to have a Prometheus instance scraping selected time series from other instances, effectively serving as a higher-level aggregating instance. This can happen in a hierarchical fashion, with each layer aggregating metrics from lower-level instances into larger-encompassing time series, or in a cross-service pattern, where a few metrics are selected from instances in the same level for federation so that some recording and alerting rules become possible. For example, you could collect data for service throughput or latency in each...