The Prometheus Data Model and PromQL

With Prometheus deployed, let’s explore a little more about what it’s doing under the hood. You’re likely already somewhat familiar with Prometheus, but you may not know the intricacies of how it works. It’s something I’m still learning every day and I dare say that Prometheus’s code base has grown to the size where there may not be a single person or maintainer who understands the ins and outs of how every piece of it works. Nevertheless, there’s still plenty to be learned; the more you understand a system, the better you can run and maintain it.

The core of the Prometheus system is its TSDB. Most of the work that Prometheus does is centered around either getting data into the TSDB or reading data out of it. But TSDBs are complicated topics that are ever-changing and evolving. Even Prometheus underwent a major TSDB rewrite when it went from version 1 to version 2. So, we’ll keep our inquiry...

Technical requirements

Presuming you installed Prometheus using the guide in the previous chapter, you will only need the following program to be installed and available in your PATH so that you can connect to Prometheus:

• kubectl: https://kubernetes.io/docs/tasks/tools/#kubectl

Please refer to this book’s GitHub repository for additional resources: https://github.com/PacktPublishing/Mastering-Prometheus

Prometheus’s data model

Prometheus’s data model is – in my opinion – straightforward. It has three main data types that are all built on each other: metrics, time series, and samples.

Metrics

Metrics are the fundamental Prometheus data type. After all, Prometheus is a metrics-focused observability tool.

Every metric has – at a minimum – a name that identifies it. This name can contain letters, digits, underscores, and/or colons. To be valid, it must match the [a-zA-Z_:][a-zA-Z0-9_:]* regex.

Additionally, metrics may also include a HELP text and a TYPE field. These are optional but highly recommended to improve usability.

A full metric that’s been exposed to Prometheus may look like this:

# HELP mastering_prometheus_readers_total Number of readers of this book
# TYPE mastering_prometheus_readers_total counter
mastering_prometheus_readers_total 123467890

The HELP line just provides some arbitrary information on what...

Prometheus’ TSDB

Prometheus’s TSDB is at the core of all that Prometheus does. It’s been in its third iteration since 2017 and continues to be regularly updated with new optimizations and features. However, its core functionality and format have largely remained the same.

Understanding how the TSDB works will enable you to understand Prometheus’s internals in a new way that can inform how you maintain and use Prometheus. The core parts of the TSDB that we’ll look at are its head block, write-ahead log (WAL), and data format (blocks, chunks, and indices), as well as how compaction works.

Head block

The head block in Prometheus is the first stop for samples being scraped and stored in Prometheus’ TSDB. It is an in-memory block (more on the specifics of a block later) containing various chunks for each of the different time series you’re collecting. This block can contain millions of chunks, depending on the number of scrape targets...

PromQL basics

With all this talk of how data gets into Prometheus’s database and how it’s stored there, I reckon it’s about time we look at how we get that data out and make use of it. To do this, Prometheus has its own query language: PromQL.

PromQL is a vast topic, and you can craft some cool queries with it, especially concerning alerting (covered in Chapter 5).

Note

Most PromQL queries provided as examples will work in your Prometheus environment from Chapter 2!

Feel free to tweak them and experiment with them as you go along by using kubectl port-forward svc/mastering-prometheus-kube-prometheus 9090 and opening http://localhost:9090 in your browser to connect to Prometheus.

Syntax overview

PromQL is pretty straightforward and does not have a litany of different data types you need to concern yourself with. The matching operators for label filters are intuitive, and there are only four possible types that PromQL expressions can be evaluated...

Summary

In this chapter, we ramped up our learning by looking at Prometheus’s data model, reviewing some PromQL basics, and diving deep into the internals of Prometheus’s TSDB. I learned a lot and I hope you did, too.

We’ll continue to build on our PromQL skills throughout this book, especially in our chapter on alerting. Before we get to that, though, we’ll look at one of Prometheus’s “killer” features that make it so well suited to containerized and cloud-native environments: service discovery.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Ganesh Vernekar’s outstanding blog post series on the Prometheus TSDB: https://ganeshvernekar.com/blog/prometheus-tsdb-the-head-block/
• Prometheus documentation: https://prometheus.io/docs/
• The original Prometheus TSDB write-up by its author: https://web.archive.org/web/20210622211933/https://fabxc.org/tsdb/