Chapter 10: Alerting and Notifications in Grafana

Alerting and notification features are an important part of any Internet of Things (IoT) platform. Fortunately, Grafana includes both. The alerting system included in Grafana allows you to specify very precise alert conditions based on your data. On the other hand, the notifications integrations let you send messages across many communication channels.

In this chapter, we will see all the options for generating alerts in Grafana. You will also learn how to integrate different notification systems.

In this chapter, you will see the following:

• How Grafana manages alerts
• Building numerical-based alerts
• Configuring notifications
• Connecting alerts and notifications

With Grafana, you can configure alerts and notifications in a matter of minutes. On the other hand, alert conditions can be as complex as you need. In this chapter, you will obtain the tools for creating your alerts and connecting to internal and...

Technical requirements

To get the most out of this chapter, you will need the following:

• A running instance of Grafana
• A data source with numerical and categorical data
• An email service
• External notification services, such as Slack, Telegram, Discord, or general Webhooks

How Grafana manages alerts

From version 8.0, Grafana manages alerts in a centralized way, so you can create, manage, and view alerts in the same place, as you can see in the following screenshot:

Figure 10.1 – Alerting manager in Grafana

This alerting system is enabled by default.

When the alerting manager is enabled, you can do the following:

• Create alert rules.
• View current rules and manage their state.
• View the state and health of rules.
• Add or modify contact points (notification systems).
• Add or modify notification policies.
• Add or modify silences.

Grafana uses Alertmanager from Prometheus for managing alerts. Alertmanager is embedded in Grafana, so you don’t have to install it separately. If you want to learn about Prometheus Alertmanager, you can go to the following page: https://prometheus.io/docs/alerting/latest/alertmanager/.

Now, let’s see how you can use Alertmanager.

Using Alertmanager...

Building numerical-based alerts

Grafana manages alerts using queries in the following data sources:

• All the data sources integrated by Grafana Labs: InfluxDB, Graphite, Prometheus, Loki, Elasticsearch, Google Cloud Monitoring, Cloudwatch, Azure Monitor, MySQL, PostgreSQL, MSSQL, OpenTSDB, and Oracle. In all the cases, the alerting must be enabled.
• All the community-developed data sources that have alerting enabled.

Besides data sources alerting, Grafana has its internal alerting metrics. You can see all the internal metrics in the following table:

Table 10.1 – Internal metrics for Grafana alerts

Time-series and tabular data

In this book, we have focused mainly on time-series data. When you use time-series data, each record must be reduced to a single number, so alert rules can be applied.

On the other hand, if you have tabular data, you only have to compare the value obtained by the query.

You can process alerts in tabular...

Notifications in Grafana

In the newest versions of Grafana, notifications are implemented using contact points. These contact points determine how the users will be notified when an alert triggers.

You can have several contact point types for the same contact point. For instance, you may have email and Slack. If an alert is triggered, all the contact point types included in the contact point will be used. You can also customize each notification message for every contact point type.

Let’s see how you can create a contact point.

Adding a contact point

To add a contact point, you have to go to the Contact points tab on the Alerting page, as you can see in Figure 10.6.

Once there, you can perform the following steps:

1. Click on the New contact point button.
2. Give a name to your new contact point.
3. From the drop-down list, select Alertmanager.
4. Click on the New contact point type button.
5. In the Contact point type section, select the one you want...

Connecting alerts and notifications

So far, you have learned how to build alerts and manage them. Also, you have seen how to create contact points to deliver alert notifications to the users.

In this section, you will learn how to connect both parts. So, you will be able to send notifications of specific alerts, in a specific way, to specific users. You can do this in Grafana using notification policies.

By using notification policies, you can define how the alerts will be routed to the contact points. These policies are built on a tree structure, where each policy can have one or more child policies.

With the exception of the root policy, all the policies can match specific alert labels. Every time an alert occurs, it is evaluated by the root policy first, and then by each child policy. If you enable the Continue matching subsequent sibling nodes option for a policy, the evaluation will continue even after one or more matches.

Policies have a hierarchical behavior. Any...

Summary

In this chapter, you have learned about the alerting and notification systems included in Grafana.

First, you learned how to create and manage alerts using the Alertmanager embedded in Grafana.

Then, you learned about notifications, contact points, and contact point types.

Finally, you learned how to connect alerts and contact points using notification policies. Also, you have seen how to silence and mute notifications.

The next chapter starts a new part of this book: Integrating Grafana with other Platforms. In Chapter 11, Using Grafana with Prometheus, you will learn how to use Grafana in combination with Prometheus.