Chapter 6: Alerting on ML Analysis

The previous chapter (Chapter 5, Interpreting Results) explained in depth how anomaly detection and forecasting results are stored in Elasticsearch indices. This gives us the proper background to now create proactive, actionable, and informative alerts on those results.

At the time of writing this book, we find ourselves at an inflection point. For several years, Elastic ML has relied on the alerting capabilities of Watcher (a component of Elasticsearch) as this was the exclusive mechanism to alert on data. However, a new platform of alerting has been designed as part of Kibana (and was deemed GA in v7.11) and this new approach will be the primary mechanism of alerting moving forward.

There are still some interesting pieces of functionality that Watcher can provide that are not yet available in Kibana alerting. As such, this chapter will showcase the usage of alerts using both Kibana alerting and Watcher. Depending on your needs, you can decide...

Technical requirements

The information in this chapter will use the Elastic Stack as it exists in v7.12.

Understanding alerting concepts

Hopefully, without running the risk of being overly pedantic, a few declarations can be made here about alerting and how certain aspects of alerting (especially with respect to anomaly detection) are extremely important to understand before we get into the mechanics of configuring those alerts.

Anomalies are not necessarily alerts

This needs to be explicitly said. Often, users who first embrace anomaly detection feel compelled to alert on everything once they realize that you can alert on anomalies. This is potentially a really challenging situation if anomaly detection is deployed across hundreds, thousands, or even tens of thousands of entities. Anomaly detection, while certainly liberating users from having to define specific, rule-driven exceptions or hardcoded thresholds from alerts, also has the potential to be deployed broadly across a lot of data. We need to be cognizant that detailed alerting on every little anomaly could be potentially...

Building alerts from the ML UI

With the release of v7.12, Elastic ML changed its default alert handler from Watcher to Kibana alerting. Prior to v7.12, the user had a choice of accepting a default watch (an instance of a script for Watcher) if alerting was selected from the ML UI, or the user could create a watch from scratch. This section will focus on the new workflow using Kibana alerting as of v7.12, which offers a nice balance of flexibility and ease of use.

To create a working, illustrative example of real-time alerting, we will contrive a scenario using the Kibana sample web logs dataset that we first used in Chapter 3, Anomaly Detection.

The process outlined in this section will be as follows:

1. Define some sample anomaly detection jobs on the sample data.
2. Define two alerts on two of the anomaly detection jobs.
3. Run a simulation of anomalous behavior, to catch that behavior in an alert.

Let's first define the sample anomaly detection jobs.

...

Creating an alert with a watch

Prior to version 7.12, Watcher was used as the mechanism to alert on anomalies found by Elastic ML. Watcher is a very flexible native plugin for Elasticsearch that can handle a number of automation tasks and alerting is certainly one of them. In versions 7.11 and earlier, users could either create their own watch (an instance of an automation task in Watcher) from scratch to alert on anomaly detection job results or opt to use a default watch template that was created for them by the Elastic ML UI. We will first look at the default watch that was provided and then will discuss some ideas around custom watches.

Understanding the anatomy of the legacy default ML watch

Now that alerting on anomaly detection jobs is handled by the new Kibana alerting framework, the legacy watch default template (plus a few other examples) are memorialized in a GitHub repository here: https://github.com/elastic/examples/tree/master/Alerting/Sample%20Watches/ml_examples...

Summary

Anomaly detection jobs are certainly useful on their own, but when combined with near real-time alerting, users can really harness the power of automated analysis – while also being confident about getting only alerts that are meaningful.

After a practical study of how to effectively capture the results of anomaly detection jobs with real-time alerts, we went through a comprehensive example of using the new Kibana alerting framework to easily define some intuitive alerts and we tested them with a realistic alerting scenario. We then witnessed how an expert user can leverage the full power of Watcher for advanced alerting techniques if Kibana alerting cannot satisfy the complex alerting requirements.

In the next chapter, we'll see how anomaly detection jobs can assist not only with alerting on important key performance indicators but also how Elastic ML's automated analysis of a broad set of data within a specific application context is the means to achieving...