Preface

Since the Prometheus project was first announced to the world in January 2015, it has rapidly become the de facto modern monitoring solution. Open source projects such as Kubernetes expose Prometheus metrics by default, cloud providers sell “managed” Prometheus services, and it even has its own yearly conference. However, in my personal journey to learn and understand Prometheus deeper, I came to a saddening realization. There are a plethora of blog posts, books, and tutorials focused on the basics of Prometheus, but few to no readily available resources that cover running Prometheus at scale.

For that information, I found myself needing to turn to conference talks or trying to extrapolate how others do it by reading through GitHub issues on the Prometheus repository, questions on the Prometheus mailing list, and conversations in the official Prometheus Slack channel. Those learnings – coupled with years of personal experience – have gone into this book as an endeavor to begin filling that void.

Perhaps one of the limiting factors of existing content is that it tends to focus purely on Prometheus itself, omitting the larger ecosystem surrounding it. However, to run Prometheus at scale, it quickly becomes necessary to build on top of Prometheus to extend it. Instead of Prometheus being the destination, it provides a foundation.

We will still cover Prometheus itself in depth. We will see how to get the most out of it, implement best practices, and – perhaps most critically – develop a deeper understanding of how Prometheus’s internals work.

In addition to Prometheus itself, though, we’ll also look at how to operate and scale Prometheus. We’ll learn how to debug Prometheus through Go’s developer tools, how to manage hundreds (or thousands) of Prometheus rules without losing your mind, how to connect Prometheus to remote storage solutions, how to run dozens of highly available Prometheus instances while maintaining a global query view, and much, much more.

Who this book is for

This book is primarily intended for readers with a preexisting, basic knowledge of Prometheus. You’re probably already running Prometheus, know what the Node Exporter and Alertmanager are, and can write PromQL queries without too much help.

If this is your first introduction to Prometheus, that’s OK, too! The content should still be generally accessible, but you may – at times – wish to seek out additional resources to explain some of the basics that are glossed over.

Regardless of your experience level, this book is targeted at operators of Prometheus. If you’re a developer who uses a Prometheus environment operated by someone else, there are still chapters that will be of interest to you, but there are others that are unlikely to be applicable to your responsibilities.

What this book covers

Chapter 1, Observability, Monitoring, and Prometheus, gives a brief overview of the history of modern monitoring systems, establishes common observability terminology and concepts, and looks at Prometheus’ role within observability.

Chapter 2, Deploying Prometheus, goes through the process of deploying Prometheus to Kubernetes and provides the lab environment that we will use throughout the rest of the book.

Chapter 3, The Prometheus Data Model and PromQL, dives deep into the technical specifics of how Prometheus – and especially its Time Series DataBase (TSDB) – works, along with an overview of how the Prometheus Query Language (PromQL) works.

Chapter 4, Using Service Discovery, goes into the details of how to use dynamic service discovery in Prometheus, including how to build your own service discovery providers.

Chapter 5, Effective Alerting with Prometheus, focuses on making Prometheus alerting reliable and testable, along with making the most of the Alertmanager.

Chapter 6, Advancing Prometheus: Sharding, Federation, and HA, is where we begin to look at scaling Prometheus past a single Prometheus server and into reliable, distributed deployments.

Chapter 7, Optimizing and Debugging Prometheus, explores how to leverage Go tools to debug the Prometheus application and how to tune Prometheus for optimum performance.

Chapter 8, Enabling Systems Monitoring with the Node Exporter, looks in depth at the most commonly deployed Prometheus exporter to understand all that it can do.

Chapter 9, Utilizing Remote Storage Systems with Prometheus, examines Grafana Mimir and VictoriaMetrics as two options that Prometheus can send data to for long-term storage, global query view, multi-tenancy support, and more.

Chapter 10, Extending Prometheus Globally with Thanos, comprehensively explores all components of the Thanos project to see how they can be used to extend the functionality of Prometheus, enable high availability, and provide for nearly unlimited retention of metrics.

Chapter 11, Jsonnet and Monitoring Mixins, introduces the Jsonnet programming language as a tool to simplify the management of Prometheus rules at scale. Additionally, we see how the Monitoring Mixins project from various Prometheus maintainers and contributors makes use of Jsonnet to provide configurable, reusable alerts and dashboards for various systems and software.

Chapter 12, Utilizing Continuous Integration (CI) Pipelines with Prometheus, takes a practical look at how you can manage your Prometheus configuration and alerts in Git and perform a variety of automated tests to them to ensure they are valid and conform to expectations.

Chapter 13, Defining and Alerting on SLOs, explores how Prometheus can be used to define, measure, and alert on Service Level Objectives (SLOs), including through the use of open source tools such as Pyrra and Sloth that make it easy to implement best-practice SLO alerting.

Chapter 14, Integrating Prometheus with OpenTelemetry, takes a look at the OpenTelemetry project – its history, its future, and how Prometheus integrates with it.

Chapter 15, Beyond Prometheus, brings us full circle back to our initial discussion about observability and provides ideas on where to go next in building out your observability suite.

To get the most out of this book

You will need to have some level of hands-on experience with Prometheus, a basic understanding of server administration, and – ideally – some prior experience with Kubernetes. The lab environment used throughout this book is based on Kubernetes, but all commands used to interact with Kubernetes are explicitly stated. Therefore, prior Kubernetes knowledge is not required, but it will certainly aid in deeper understanding and enable further experimentation outside of what is covered in the book.

Software/hardware covered in the book	Operating system requirements
Prometheus	Windows, macOS, or Linux
Thanos	Windows, macOS, or Linux
OpenTelemetry	Windows, macOS, or Linux
Kubernetes	Windows, macOS, or Linux

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Mastering-Prometheus. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Mount the downloaded WebStorm-10*.dmg disk image file as another disk in your system.”

A block of code is set as follows:

>>> hashB = int(md5(SEPARATOR.join(targetB).encode("utf-8")).hexdigest(), 16)
>>> hashB
139861250730998106692854767707986305935
>>> print(f"{targetA} % {MOD} = ", hashA % MOD)
['app=nginx', 'instance=node2'] % 2 =  0
>>> print(f"{targetB} % {MOD} = ", hashB % MOD)
['app=nginx', 'instance=node23'] % 2 =  1

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

alerting:
  alert_relabel_configs:
    - regex: prometheus_replica
      action: labeldrop

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “Select System info from the Administration panel.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Mastering Prometheus, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below

https://packt.link/free-ebook/978-1-80512-566-2

2. Submit your proof of purchase
3. That’s it! We’ll send your free PDF and other benefits to your email directly