Utilizing Remote Storage Systems with Prometheus

On its own, Prometheus can take you pretty far in establishing your monitoring stack and making systems more observable with metrics. Nevertheless, Prometheus is intentionally limited in its feature set and functionality; this is what helps keep it simple and flexible. With Prometheus’s huge community support, several other open source projects have arisen to augment and extend Prometheus’ base features. Throughout the remainder of this book, we’ll be focusing on these projects and how they can help you make the most of your Prometheus environment.

To begin with, we’re going to look at Prometheus’ remote storage system, which contains methods for both remotely reading and remotely writing Prometheus data. We’ll be looking at two popular open source projects that leverage these APIs, including how to deploy them and hook them up to your Prometheus instances.

In this chapter, we’re...

Technical requirements

For this chapter, we’ll continue building off of the Prometheus environment we deployed to Kubernetes in Chapter 2. Consequently, you’ll need the two following tools installed:

• kubectl: https://kubernetes.io/docs/tasks/tools/#kubectl
• helm: https://helm.sh/docs/intro/install/

The code used in this chapter is available at https://github.com/PacktPublishing/Mastering-Prometheus.

Understanding remote write and remote read

Prometheus is purposely limited in the functionality it endeavors to implement. Notably, its storage is tied to a local filesystem, and there is no clustering or replication support for Prometheus instances. When running Prometheus at a sufficient scale, you will inevitably encounter the desire to scale Prometheus beyond these limitations. Rather than constrain you to only using Prometheus, Prometheus has built-in functionality to integrate with other metrics storage systems that are more featureful (but also more complex).

Prometheus does this by exposing two remote storage APIs—one for reading data from a storage system and one for writing data to it. Both APIs are exposed via HTTP, use protocol buffers (protobufs) for communication, and use “snappy” compression for both requests and responses.

Remote write is certainly the more popular of the two since it involves sending data to a larger metrics system, which...

Using VictoriaMetrics

VictoriaMetrics is one of the most popular and established options for the remote storage of Prometheus metrics. In fact, some people and projects use VictoriaMetrics as a drop-in replacement for Prometheus since it supports directly scraping targets via its vmagent component and is mostly compatible with the Prometheus configuration file (it lacks support for the remote_write, remote_read, rule_files, and alerting sections).

One of the primary reasons that people choose VictoriaMetrics as their remote storage destination is its resource efficiency. In a head-to-head benchmark against Prometheus, VictoriaMetrics boasts seven times the reduction in disk space (~0.3 bytes per sample vs. ~2.1 bytes) and five times the reduction in memory (~4.3GB of RAM vs. ~23GB) for the same amount of time series and samples (2.8 million active time series and 280,000 samples/sec) in their benchmark, published at https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark...

Using Grafana Mimir

Grafana Labs’ Mimir project is a relative newcomer to the remote storage Prometheus ecosystem. Announced in 2022, Mimir is the spiritual successor to the popular Cortex project, which is a Cloud Native Computing Foundation (CNCF)-sponsored project (as with Prometheus, Thanos, Kubernetes, and many more). It aims to be the most scalable and performant option available for time series storage.

Important note

Is Cortex dead? Cortex is still under active development and is used extensively by numerous companies. If Mimir intrigues you, but its AGPLv3 license is off-putting, try out Cortex! It’ll still be a mostly similar experience and architecture.

Note that Mimir does not mention being the most efficient in its goals. If you approach Mimir thinking that you are going to see the same massive gains in memory and storage efficiency as VictoriaMetrics, you will be disappointed. However, you still can expect its storage efficiency to be about as good...

Summary

In this chapter, we learned about how Prometheus can integrate with a variety of remote storage systems through the remote read and remote write protocols. We made that practical by deploying two popular remote storage backends: VictoriaMetrics and Mimir. Both can be used to extend Prometheus far beyond what simple federation is capable of in order to support advanced use cases involving the multi-tenancy and long-term storage of Prometheus metrics.

In our next chapter, we’re going to look at another popular open source project that integrates with Prometheus: Thanos. It integrates with remote storage protocols but also can be used for a whole lot more.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Prometheus agent: https://prometheus.io/blog/2021/11/16/agent/
• Remote write configuration and tuning:
  • https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
  • https://prometheus.io/docs/practices/remote_write/#remote-write-tuning
• Prometheus “stringlabels”: https://prometheus.io/blog/2023/03/21/stringlabel/
• VictoriaMetrics:
  • https://docs.victoriametrics.com
  • https://valyala.medium.com/prometheus-vs-victoriametrics-benchmark-on-node-exporter-metrics-4ca29c75590f
  • https://docs.victoriametrics.com/Articles.html#benchmarks
  • https://victoriametrics.com/plans-features/
  • https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e
• Mimir:
  • https://grafana.com/blog/2022/03/30/announcing-grafana-mimir/
  • https://victoriametrics.com/blog/mimir-benchmark/
  • https://grafana.com/docs/helm-charts/mimir-distributed...