Chapter 8. Anomaly Detection in Kibana 5.0

In September 2016, Elastic announced the acquisition of Prelert, now called Machine Learning, a behavioral analytics company. Prelert combines an anomaly detection engine, Elasticsearch for storing the analysis, and Kibana for visualizing the analysis.

The anomaly detection engine brings unsupervised machine learning capabilities to the Elastic Stack so that Prelert is able to learn from the data as it ingests them, and can highlight events that deviate from expectations.

In this chapter we'll explore the following:

• Applying the use case of Prelert to find a solution in anomaly detection

• Using Prelert and Kibana for operational analytics

• Leveraging Timelion, X-Pack alerting, and reporting features to visualize and be apprised of anomalies

Tip

As a disclaimer, the version of Prelert used in this chapter is an exclusive preview of what Prelert will look like in the upcoming GA version. It's not a public version. At the time of writing this chapter, the current...

Understanding the concept of anomaly detection

In this section, we'll try to summarize how Prelert solves the challenge of anomaly detection by first understanding why data visualization is a sufficient medium when it comes to pointing out an anomaly, and then we'll see why traditional alerting systems cannot be used at scale for anomaly detection.

Understanding human limits with regard to data visualization

Anomaly detection is the art of detecting things that shouldn't occur, or that differ from normal occurrences. Anomaly detection is the general name given to a statistical modeling technique used to identify unusual patterns in time-based events.

If we take the following dashboard, we can see different things happening:

IT ops dashboard with potential anomalies

In the preceding screenshot, we can see a significant drop in the first graph (point 1). This looks suspicious, and may indicate a problem. Now, compared with the rest of the charts alongside it, we see that the increases in points...

Using Prelert for operational analytics

In this section, we'll use what we learned in Chapter 5, Metric Analytics with Metricbeat and Kibana 5.0 and apply it to Prelert. The idea is to use Metricbeat to generate system data and analyze the CPU utilization, as well as to detect anomalies. We'll run Metricbeat on our machines; you can do the same on a different machine, if you have some on Amazon, for instance. Wherever you do it, we'll also run a stress tool to generate CPU utilization, just to facilitate the demo so that we are sure that we have the anomalies.

The first thing to do is download Metricbeat, install it, and import Kibana dashboards, as shown in Chapter 5, Metric Analytics with Metricbeat and Kibana 5.0; refer to this chapter for more details. Once installed, run Metricbeat and start generating data.

Setting up Prelert

At the time of writing, only four weeks have passed since Prelert was acquired by Elastic, which means that the integration of Prelert in Elastic Stack is still...

Combining Prelert, alerting, and Timelion

Prelert detects anomalies in data indexed in Elasticsearch, stores its results in Elasticsearch, but also provides out of the box dashboards to explore and understand anomalies. The Elastic stack provides a holistic platform for data analysis, in which we can just pick products to extend our anomaly detection experience. X-Pack alerting is the first choice as it could consume Prelert results to trigger relevant and accurate alerts. Timelion is also a fantastic choice to correlate the anomaly detection result to source data by using the statistics functions and customization features that it offers.

As said earlier, Prelert exposes a REST API that allows you to manage a job and get the result of the analysis.

Job details and endpoints

The preceding image details an Endpoint links section in which the REST APIs that we are going to use for alerting are listed. All APIs are documented at http://www.prelert.com/docs/products/latest/engine_api_reference...

Summary

In this chapter, we have seen why traditional approaches to anomy detection quickly converge to their limit, whether from a human point of view (because of the amount of information to digest); or from the technical point of view where traditional statistical methodologies generate false positives or true negatives. Then we leverage the dataset and use cases build in the previous chapter to illustrate how Kibana can be used for anomaly detection based on the unsupervised machine learning feature that Machine Learning brings to the Elastic Stack.

In the next and final chapters, we'll tackle the subject of Kibana custom plugin creation by first setting up the development environment and then implementing the plugin.