Observability, Monitoring, and Prometheus

Observability and monitoring are two words that are often used synonymously but carry important distinctions. While this book is not focused on academic definitions and theories surrounding observability, it’s still useful to distinguish between observability and monitoring because it will provide you with a framework to get in the right mindset when thinking about how Prometheus works and what problems it solves. A screw and a nail can both hang a picture, and you can bang a screw into a wall with a hammer, but that doesn’t make it the best tool for the job. Likewise, with Prometheus, I’ve seen many people fall into the trap of trying to use Prometheus to cover all of their observability and monitoring needs – when you have a hammer, everything looks like a nail. Instead, let’s identify where Prometheus shines so that we can use it to its full effect throughout the rest of this book.

In this chapter, we...

A brief history of monitoring

In the beginning, there was Nagios… or, at least, so the story goes. Monitoring as we know it took off in the late 1990s and early 2000s with the introduction of tools such as Nagios, Cacti, and Zabbix. Sure, some things existed before that that focused on network monitoring such as Multi Router Traffic Grapher (MRTG) and its offshoot, rrdtool, but system monitoring – including servers – found its stride with Nagios. And it was good… for a time.

Nagios (and its ilk) served its purpose and – if your experience is anything like mine – it just won’t seem to go away. That’s because it does a simple job, and it does it fairly well. Let’s look a little closer at it, the philosophy it embodies, and where it differs from Prometheus.

Nagios

Early monitoring tools such as Nagios were check-based. You give it a script to run with some basic logic and it tells you whether things are good, bad...

Introduction to observability concepts

Observability both as a word and as a discipline is not unique to technology. The term is derived from control theory, which is traditionally more rooted in physical engineering disciplines such as robotics and nuclear engineering. It is, in essence, the ability to surmise the health of a system by observing its inputs and outputs. In nuclear engineering, you put in uranium and water, and you receive heat and steam. In software engineering, you put in an end user and an API call, and you receive a Jira ticket about how your API isn’t working. Err… well, hopefully not if your observability is doing its job.

Observability in systems engineering and software is primarily informed by and achieved with a handful of important telemetry signal types. You may have heard them referred to as “the three pillars of observability,” but that terminology has since fallen out of fashion as it elevates the act of gathering telemetry...

Prometheus’s role in observability

Prometheus is objectively pretty great at what it does, but can we make a system fully observable with just Prometheus? Unfortunately, the answer to that question is no. Prometheus’s strength is also its weakness – it’s singularly focused on one thing: metrics.

Prometheus is only focused on the metrics aspect of an observable system. It is purpose-built to efficiently store numeric time series of varying types in a simple format. To the extent to which it interoperates with other observability signals, it is only to provide a link or bridge to some other purpose-built system.

However, Prometheus also provides some of the highest-value data that you can collect. It’s likely your go-to data source in tools such as Grafana to visualize how your systems are performing. Logging and tracing systems can certainly provide more detailed data, but to get them to provide that same level of value in analyzing trends, you...

Summary

In this chapter, we learned the abridged history of monitoring and observability, what observability is, and how Prometheus contributes to observability through metrics. With this new (or refreshed) frame of mind, we can approach our utilization of Prometheus in a way that maximizes its usefulness without trying to use it as a silver bullet to solve all our problems.

In the next chapter, we’ll cover deploying a Prometheus environment that we’ll use as the foundation that we build upon throughout the remainder of this book.

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• Fast and flexible observability with canonical log lines: https://stripe.com/blog/canonical-log-lines