Chapter 10: Outlier Detection

In the first section of this book, we discussed anomaly detection in depth, a feature that allows us to detect unusual behavior in time series data in an unsupervised fashion. This works well when we want to detect whether one of our applications is experiencing unusual latency at a particular time or whether a host on our corporate network is transmitting an unusual number of bytes.

In this chapter, we will learn about the second unsupervised learning feature in the Elastic Stack: outlier detection, which allows us to detect unusual entities in non-time series-based indices. Some interesting applications of outlier detection could involve, for example, detecting unusual cells in a tissue sample, investigating unusual houses, or areas in a local real estate market and catching unusual binaries installed on your computer.

The outlier detection functionality in the Elastic Stack is based on an ensemble or a grouping of four different outlier detection...

Technical requirements

The material in this chapter relies on using Elasticsearch version 7.9 or above. The figures in this chapter have been generated using Elasticsearch 7.10. Code snippets and code examples used in this chapter are under the chapter10 folder in the book's GitHub repository: https://github.com/PacktPublishing/Machine-Learning-with-Elastic-Stack-Second-Edition.

Discovering how outlier detection works

Outlier detection can offer insights into datasets by discovering which points are different or unusual, but how does outlier detection in the Elastic Stack work? To understand how outlier detection functionality can be constructed, let's start by thinking conceptually about how you would design the algorithm, and then see how our conceptual ideas can be formalized into the four separate algorithms that make up the outlier detection ensemble in Elasticsearch.

Suppose for a second that we have a two-dimensional set of weight and circumference measurements...

Applying outlier detection in practice

In this section, we will take a look at a practical example of outlier detection using a public dataset describing the physicochemical properties of wine. This dataset is available for download from the University of California Irvine (UCI) repository (https://archive.ics.uci.edu/ml/datasets/wine+quality).

The wine dataset is composed of two CSV files: one describing the physicochemical properties of white wine, the other those of red wine. In this walk-through, we will be focusing on the white wine dataset, but you are welcome to use the data for red wine as well since most of the steps described in this chapter should be applicable to both.

First let's import the dataset into our Elasticsearch cluster using the Data Visualizer tool, which you can find under the Machine Learning app in Kibana.  We will make an index for the white wine dataset and call it winequality-white:

Figure 10.7 – The...

Evaluating outlier detection with the Evaluate API

In the previous section, we touched on the fact it can be hard for a user to know how to set the threshold for outlier scores in order to group the data points in the dataset into normal and outlier categories. In this section, we will show how to approach this issue if you have a labeled dataset that contains, for each point, the ground truth values that record whether the point is an outlier. Before we dive into the practical demonstration, let's take a moment to understand some key performance metrics that are used in evaluating the performance of the outlier detection algorithm.

One of the simplest ways we can measure the performance of the algorithm is to compute the number of data points that it correctly predicted as outliers; in other words, the number of true positives (TPs). In addition, we also want to know the number of true negatives (TNs): how many normal data points were correctly predicted as normal. By extension...

Hyperparameter tuning for outlier detection

For the more advanced user, the Data Frame Analytics wizard offers an opportunity to configure and tune hyperparameters – various knobs and dials that fine-tune how the outlier detection algorithm works. The available hyperparameters are displayed in Figure 10.17. For example, we can direct the outlier detection job to use only a certain type of outlier detection method instead of the ensemble, to use a certain value for the number of nearest neighbors that are used in the computation in the ensemble, and to assume that a certain portion of the data is outlying.

Please note that while it is good to play around with these settings to experiment and get a feel for how they affect the final results, if you want to customize any of these for a production usecase, you should carefully study the characteristics of your data and have an awareness of how these characteristics will interact with your chosen hyperparameter settings. More...

Summary

To conclude the chapter, let's remind ourselves of the main features of the second unsupervised learning feature in the Elastic Stack: outlier detection. Outlier detection can be used to detect unusual data points in single or multidimensional datasets.

The algorithm is based on an ensemble of four separate measures: two distance-based measures based on kth-nearest neighbors and two density-based measures. The combination of these measures captures how far a given data point is from its neighbors and from the general mass of data in the dataset. This unusualness is captured in a numerical outlier score that ranges from 0 to 1. The closer a given data point scores to 1, the more unusual it is in the dataset.

In addition to the outlier score, for each feature or field of a point, we compute a quantity known as the feature influence. The higher the feature influence for a given field, the more that field is responsible for a given point being unusual. These feature...