Defining SLOs with Prometheus data

Prometheus metrics are great for SLIs since they are generally specific enough to pick out individual aspects of a service’s performance but still generic enough to be aggregated into an overarching view of how that service is performing. Once you understand the patterns of defining SLOs with Prometheus, the most challenging part is really just identifying which of your hundreds of thousands (or millions) of metrics are best suited to being the SLIs for SLOs.

Once you have your SLI(s) picked out, the SLOs in Prometheus are typically defined in recording rules. Since SLIs will normally need to be aggregated across multiple instances or dimensions, it’s simpler to precompute their values instead of running their queries over long time ranges. I also recommend keeping the naming of your SLO recording rules consistent so that it is easy and intuitive to query for SLO metrics across services, for example, using the same resultant series...