Understanding and Extending Alertmanager

Alerting is a critical component in any monitoring stack. In the Prometheus ecosystem, alerts and their subsequent notifications are decoupled. Alertmanager is the component that handles these alerts. In this chapter, we'll be focusing on converting alerts into useful notifications using Alertmanager. From reliability to customization, we'll delve into the inner workings of the Alertmanager service, providing the required knowledge to configure, troubleshoot, and customize all the options available. We'll make sure that concepts such as alert routing, silencing, and inhibition are clear so that you can decide how to implement them in your own stack.

Since Alertmanager is a critical component, high availability will also be explored, and we will also explain the relationship between Prometheus and Alertmanager. We will customize...

Setting up the test environment

To work with Alertmanager, we'll be three new instances to simulate a highly available setup. This approach will allow us to not only expose the required configurations, but also validate how everything works together.

The setup we'll be using resembles the following diagram:

Figure 11.1: Test environment

Deployment

Let's begin by deploying the Alertmanager test environment:

1. To launch a new test environment, move into this chapter's path, relative to the repository root:

cd ./chapter11/

2. Ensure that no other test environments are running and spin up this chapter's environment:

vagrant global-status
vagrant up

3. You can validate the successful deployment of the test...

Alertmanager fundamentals

We covered how alerting rules work in Prometheus in Chapter 9, Defining Alerting and Recording Rules, but those, by themselves, aren't all that useful. As we mentioned previously, Prometheus delegates notification handling and routing to external systems through a Webhook-style HTTP interface. This is where Alertmanager comes in.

Alertmanager is responsible for accepting the alerts generated from Prometheus alerting rules and converting them into notifications. The latter can take any form, such as email messages, chat messages, pages, or even Webhooks that will then trigger custom actions, such as logging alerts to a data store or creating/updating tickets. Alertmanager is also the only component in the official stack that distributes its state across instances so that it can keep track of things such as which alerts were already sent and which...

Alertmanager configuration

In Chapter 9, Defining Alerting and Recording Rules, we discussed how Prometheus generates and pushes out alerts. Having also made clear the distinction between an alert and a notification, it's now time to use Alertmanager to handle the alerts that are sent by Prometheus and turn them into notifications.

Next, we'll go through the configuration required on Prometheus, along with the configuration options available in Alertmanager, so that we have notifications going out from our monitoring stack.

Prometheus configuration

There are a couple of configurations that need to be done in Prometheus so that we can start using Alertmanager. The first thing to do is configure the external labels...

Common Alertmanager notification integrations

Users and/or organizations have different requirements regarding notification methods; some might be using HipChat as a means of communication, while others rely on email, on-call usually demands a pager system such as PagerDuty or VictorOps, and so on. Thankfully, Alertmanager provides several integration options out of the box and covers most of the notification needs you might have. If not, there's always the Webhook notifier, which allows integration with custom notification methods. Next, we'll be exploring the most common integrations and how to configure them, as well as providing basic examples to get you started.

Something to keep in mind when considering integrating with chat systems is that they're designed for humans, and the use of a ticketing system is advised when thinking about low-priority alerting....

Customizing your alert notifications

For each of the available integrations, Alertmanager already includes built-in templates for their notifications. However, these can be tailored to the specific needs of the user and/or organization. Similar to the alerting rule annotations we explored in Chapter 9, Defining Alerting and Recording Rules, alert notifications are templated using the Go templating language. Let's use the Slack integration as an example and understand how the messages are constructed so that they are tailored to your needs.

Default message format

To have an idea of what a notification without any customization looks like, we're going to use a very simple example. Take the following alerting rule,...

Who watches the Watchmen?

The monitoring system is a critical component of any infrastructure. We rely on it to keep watch over everything – from servers and network devices to services and applications – and expect to be notified whenever there's a problem. However, when the problem is on the monitoring stack itself, or even on a notification provider so that alerts are generated but don't reach us, how will we, as operators, know?

Guaranteeing that the monitoring stack is up and running, and that notifications are able to reach recipients, is a commonly overlooked task. In this section, we will go into what can be done to mitigate risk factors and improve overall confidence in the monitoring system.

Meta-monitoring and cross-monitoring

...

Summary

In this chapter, we dived into the alerting component of the Prometheus stack, Alertmanager. This service was designed with availability in mind, and we had the opportunity to understand how it works, from generating better notifications to avoiding being flooded by useless ones. The notification pipeline is a very good starting point to grok the inner workings of Alertmanager, but we also went through its configuration, while providing examples to better solidify that knowledge. We were introduced to amtool and all the features it provides, such as adding, removing, and updating silences directly from the command line.

Alertmanager has several notification integrations available and we went through all of them, so you can pick and choose the ones you're interested in. Since we all want better notifications, we delved into how to customize the default notifications...

Questions

1. What happens to the notifications if there's a network partition between Alertmanager instances in the same cluster?
2. Can an alert trigger multiple receivers? What is required for that to happen?
3. What's the difference between group_interval and repeat_interval?
4. What happens if an alert does not match any of the configured routes?
5. If the notification provider you require is not supported natively by Alertmanager, how can you use it?
6. When writing custom notifications, how are CommonLabels and CommonAnnotations populated?
7. What can you do to ensure that the full alerting path is working from end to end?

Further reading

• Official Alertmanager page: https://prometheus.io/docs/alerting/alertmanager/
• Alertmanager notification guide: https://prometheus.io/docs/alerting/notifications/
• Alertmanager configuration specifics: https://prometheus.io/docs/alerting/configuration/