Chapter 3: Anomaly Detection

Anomaly detection was the original capability of Elastic ML and is the most mature, stretching its roots back to the Prelert days (before the acquisition by Elastic in 2016). This technology is robust, easy to use, powerful, and broadly applicable to all kinds of use cases for time series data.

This jam-packed chapter will focus on using Elastic ML to detect anomalies in the occurrence rates of documents/events, rare occurrences of things, and numerical values outside of expected normal operation. We will run through some simple but effective examples that will highlight both the efficacy of Elastic ML and its ease of use.

Specifically, we will cover the following:

• Elastic ML job types
• Dissecting the detector
• Detecting changes in event rates
• Detecting changes in metric values
• Understanding the advanced detector functions
• Splitting analysis along categorical features
• Understanding temporal versus population analysis
...

Technical requirements

The information in this chapter is based on the Elastic Stack as it exists in v7.10. As with all of the chapters, all the example code can be found on GitHub: https://github.com/PacktPublishing/Machine-Learning-with-Elastic-Stack-Second-Edition.

Elastic ML job types

When we start using the Elastic ML UI to configure anomaly detection jobs, we will see that there are five different job wizards that are shown:

Figure 3.1 – The Create job UI showing different configuration wizards

The existence of these different configuration wizards implies that there are different "types" of jobs. In actuality, there is really only one job type—it is just that the anomaly detection job has many options, and many of these wizards make certain aspects of that configuration easier. Everything that you may desire to configure can be done via the Advanced wizard (or the API). In fact, when Elastic ML was first released as beta in v5.4, that was all that existed. Since then, the other wizards have been added for simplicity and usability in specific use cases.

An anomaly detection job has many configuration settings, but the two most important ones are the analysis configuration and the datafeed...

Dissecting the detector

At the heart of the anomaly detection job are the analysis configuration and the detector. The detector has several key components to it:

• The function
• The field
• The partition field
• The by field
• The over field

We will go through each in turn to fully understand them all. Note that in the next few sections, however, we will often refer to the actual names of settings within the job configuration as if we were using the advanced job editor or the API. Although it is good to fully understand the nomenclature, as you progress through this chapter you will also notice that many of the details of the job configuration are abstracted away from the user or are given more "UI-friendly" labels than the real setting names.

The function

The detector function describes how the data will be aggregated or measured within the analysis interval (bucket span). There are many functions, but they can be classified into the following...

Detecting changes in metric values

Obviously, not all data being emitted from systems will be text or categorical in nature—a vast amount of it is numerical. Detecting changes in metric values over time is perfectly suited for anomaly detection because, as mentioned in Chapter 1, Machine Learning for IT, the historical paradigm of alerting on exceptions in numerical values via static thresholds has been troublesome for decades. Let's explore all that Elastic ML has to offer with respect to the functions that help you detect changes in numerical fields in your data.

Metric functions

Metric functions operate on numerical fields and return numerical values. They are perhaps the easiest of the detector functions to understand.

min, max, mean, median, and metric

These functions do exactly as you would expect: they return the minimum, maximum, average/mean, and median of all of the numerical observations for the field of interest in the bucket span.

The metric...

Understanding the advanced detector functions

In addition to the detector functions mentioned so far, there are also a few other, more advanced functions that allow some very unique capabilities. Some of these functions are only available if the ML job is configured via the advanced job wizard or via the API.

rare

In the context of a stream of temporal information (such as a log file), the notion of something being statistically rare (occurring at a low frequency) is paradoxically both intuitive and hard to understand. If I were asked, for example, to trawl through a log file and find a rare message, I might be tempted to label the first novel message that I saw as a rare one. But what if practically every message was novel? Are they all rare? Or is nothing rare?

In order to define rarity to be useful in the context of a stream of events in time, we need to agree that the declaration of something as being rare must take into account the context in which it exists. If there...

Splitting analysis along categorical features

We have seen the power of anomaly detection jobs in uncovering interesting anomalies in a single time series dataset. However, there are a few mechanisms by which the analysis can be split along a categorical field to invoke a parallel analysis across tens, hundreds, and even multiple thousands of unique entities.

Setting the split field

When using some of the job wizards (such as the Multi-metric and Population wizards), you will see an option to split the analysis:

Figure 3.23 – Splitting on a categorical field

Here, in Figure 3.23, which uses the Multi-metric wizard to build a job against the kibana_sample_data_ecommerce index, we see that the high sum function on the taxful_total_price field is being split per instance on the field called category.keyword (plus turning the Sparse data option on). In other words, the analysis will be done for every category of items in this e-commerce store (men...

Understanding temporal versus population analysis

We learned back in Chapter 1, Machine Learning for IT, that there are effectively two ways to consider something as anomalous:

• Whether or not something changes drastically with respect to its own behavior over time
• Whether or not something is drastically different when compared to its peers in an otherwise homogeneous population

By default, the former (which we'll simply call temporal analysis) is the mode used unless the over_field_name setting is specified in the detector config.

Population analysis can be very useful in finding outliers in a variety of important use cases. For example, perhaps we want to find machines that are logging more (or less) than similarly configured machines in the following scenarios:

• Incorrect configuration changes that have caused more errors to suddenly occur in the log file for the system or application.
• A system that might be compromised by malware may actually...

Categorization analysis of unstructured messages

Imagine that you are troubleshooting a problem by looking at a particular log file. You see a line in the log that looks like the following:

   18/05/2020 15:16:00 DB Not Updated [Master] Table

Unless you have some intimate knowledge about the inner workings of the application that created this log, you may not know whether the message is important. Having the database be Not Updated possibly sounds like a negative situation. However, if you knew that the application routinely writes this message, day in and day out, several hundred times per hour, then you would naturally realize that this message is benign and should possibly be ignored, because clearly the application works fine every day despite this message being written to the log file.

The problem, obviously, is one of human interpretation. Inspection of the text of the message and the reading of a negative phrase (Not Updated) potentially biases a person...

Managing Elastic ML via the API

As with just about everything in the Elastic Stack, ML can also be completely automated via API calls—including job configuration, execution, and result gathering. Actually, all interactions you have in the Kibana UI leverage the ML API behind the scenes. You could, for example, completely write your own UI if there were specific workflows or visualizations that you wanted.

Note

For more in-depth information about the anomaly detection APIs, please refer to elastic.co/guide/en/machine-learning/current/ml-api-quickref.html. The data frame analytics part of Elastic ML has a completely separate API, which will be discussed in Chapters 9 to 13.

We won't go into each API call, but we would like to highlight some parts that are worth a detour.

The obvious first API to mention is the job creation API, which allows the creation of the ML job configuration. For example, if you wanted to recreate the population analysis job shown in Figure...

Summary

We've seen that Elastic ML can highlight variations in volume, diversity, and uniqueness in metrics and log messages, including those that need some categorization first. Also, we've shown that population analysis can be an extremely interesting alternative to temporal anomaly detection when the focus is more on finding the most unusual entities. These techniques help solve the challenges we described before, where a human might struggle to recognize what is truly unusual and worthy of attention and investigation.

The skills learned in this chapter will be helpful in subsequent chapters, where we will see how ML assists in the process of getting to the root cause of complex IT problems, identifying application performance slowdowns, or when ML can assist in the identification of malware and/or malicious activity.

In the next chapter, we'll see how the expressive time series models built by anomaly detection jobs can be leveraged to forecast trends of your...