Chapter 8: Anomaly Detection in Other Elastic Stack Apps

When the first edition of this book was authored two years ago, there was no concept of other apps within the stack leveraging Elastic ML for domain-specific solutions. However, since then, Elastic ML has become a provider of anomaly detection for domain-specific solutions, providing tailor-made job configurations that users can enable with a single click.

In this chapter, we will explore what Elastic ML brings to various Elastic Stack apps:

• Anomaly detection in Elastic APM
• Anomaly detection in the Logs app
• Anomaly detection in the Metrics app
• Anomaly detection in the Uptime app
• Anomaly detection in the Elastic Security app

Technical requirements

The information in this chapter is relevant as of v7.12 of the Elastic Stack.

Anomaly detection in Elastic APM

Elastic APM takes application monitoring and performance management to a whole new level by allowing users to instrument their application code to get deep insights into the performance of individual microservices and transactions. In complex environments, this could generate a large number of measurements and poses a potentially paradoxical situation – one in which greater observability is obtained via this detailed level of measurement while possibly overwhelming the analyst who has to sift through the results for actionable insights.

Fortunately, Elastic APM and Elastic ML are a match made in heaven. Anomaly detection not only automatically adapts to the unique performance characteristics of each transaction type via unsupervised machine learning, but it can also scale to handle the possibly voluminous amounts of data that APM can generate.

While the user is always free to create anomaly detection jobs against any kind of time-series...

Anomaly detection in the Logs app

The Logs app inside of the Observability section of Kibana offers a similar view of your data as the Discover app. However, the users who appreciate more of a live tail view of their logs, regardless of the index the data is stored, will love the Logs app:

Figure 8.16 – The Logs app, part of the Observability section of Kibana

Notice that there is both an Anomalies tab and a Categories tab. Let's first discuss the Categories section.

Log categories

Elastic ML's categorization capabilities, first shown back in Chapter 3, Anomaly Detection, are applied in a generic way to any index of unstructured log data. Within the Logs app, however, categorization is employed with some more strict constraints on the data. In short, the data is expected to be in Elastic Common Schema (ECS) with certain fields defined (especially a field called event.dataset).

Note

The logs dataset from Chapter 7, AIOps and Root...

Anomaly detection in the Uptime app

The Uptime app allows simple availability and response time monitoring of services via a variety of network protocols, including HTTP/S, TCP, and ICMP:

1. Often classified as synthetic monitoring, the Uptime app uses Heartbeat to actively probe network endpoints from one or more locations:

     

   Figure 8.25 – The Uptime app in Kibana

2. If you would like to enable anomaly detection on a monitor, simply click on the monitor name to see the monitor detail. Within the Monitor duration panel, notice the Enable anomaly detection button:

     

   Figure 8.26 – Enabling anomaly detection for an Uptime monitor

3. Clicking on the Enable anomaly detection button creates the job in the background and offers the user the option to create an alert for anomalies surfaced by the job:

     

   Figure 8.27 – Creating an alert on the anomaly detection job in the Uptime app

4. Once the anomaly detection job is available, any anomalies...

Anomaly detection in the Elastic Security app

Elastic Security is truly the quintessence of a purpose-driven application in the Elastic Stack. Created from the ground up with the security analyst's workflow in mind, the comprehensiveness of the Elastic Security app could fill an entire book on its own. However, the heart of the Elastic Security app is the Detections feature in which user- and Elastic-created rules execute to create alerts when rules' conditions are met. As we'll see, Elastic ML plays a significant role in the Detections feature.

Prebuilt anomaly detection jobs

The majority of the detection rules in Elastic Security are static, but many are backed by prebuilt anomaly detection jobs that operate on the data collected from Elastic Agent or Beats, or equivalent data that conforms with the ECS fields that are applicable for each job type. To see a comprehensive list of anomaly detection jobs supplied by Elastic, view the datafeed and job configuration...

Summary

Elastic ML has clearly infiltrated many of the other apps in the Elastic Stack, bringing easy-to-use functionality to users' fingertips. This proves how much Elastic ML is really a core functionality to the Stack itself, akin to other key stack features such as aggregations.

Congratulations, you have reached the end of the first half of this book, and hopefully you feel well armed with everything that you need to know about Elastic ML's anomaly detection.

Now, we will venture into the "other side" of Elastic ML – data frame analytics – where you will learn how to bring other machine learning techniques (including supervised-based model creation and inferencing) to open up analytical solutions to a vast new array of use cases.