Appendix: Anomaly Detection Tips

As we wind down the content for this book, it occurred to us that there's still a plethora of good, bite-sized explanations, examples, and pieces of advice that didn't quite fit into sections of the other chapters. It therefore made sense to give them a home all to themselves here in the Appendix. Enjoy this potpourri of tips, tricks, and advice!

The following topics will be covered here in the Appendix:

• Understanding influencers in split versus non-split jobs
• Using one-sided functions to your advantage
• Ignoring time periods
• Using custom rules and filters to your advantage
• Anomaly detection job throughput considerations
• Avoiding the over-engineering of a use case
• Using anomaly detection on runtime fields

Technical requirements

The information in this chapter will use the Elastic Stack as it exists in v7.12.

Understanding influencers in split versus non-split jobs

You might question whether or not it is necessary to split the analysis by a field, or merely hope that the use of influencers will give the desired effect of identifying the offending entity.

Let's remind ourselves of the difference between the purpose of influencers and the purpose of splitting a job. An entity is identified by an anomaly detection job as an influencer if it has contributed significantly to the existence of the anomaly. This notion of deciding influential entities is completely independent of whether or not the job is split. An entity can be deemed influential on an anomaly only if an anomaly happens in the first place. If there is no anomaly detected, there is no need to figure out whether there is an influencer. However, the job may or may not find that something is anomalous, depending on whether or not the job is split into multiple time series. When splitting the job, you are modeling (creating...

Using one-sided functions to your advantage

Many people realize the usefulness of one-sided functions in ML, such as low_count and high_mean, to allow for the detection of anomalies only on the high side or on the low side. This is useful when you only care about a drop in revenue or a spike in response time.

However, when you care about deviations in both directions, you are often inclined to use just the regular function (such as count or mean). However, on some datasets, it is more optimal to use both the high and low versions of the function as two separate detectors. Why is this the case and under what conditions, you might ask?

The condition where this makes sense is when the dynamic range of the possible deviations is asymmetrical. In other words, the magnitude of potential spikes in the data is far, far bigger than the magnitude of the potential drops, possibly because the count or sum of something cannot be less than zero. Let's look at the following screenshot...

Ignoring time periods

Often, people ask how they can get ML to ignore the fact that a certain event has occurred. Perhaps it was an expected maintenance window, or perhaps something was broken within the data ingest pipeline and data was lost for a few moments. There are a few ways that you can get ML to ignore time periods, and for distinction, we'll separate them into two groups:

• A known, upcoming window of time
• An unexpected window of time that is discovered only after the fact

To illustrate things, we'll use a single-metric count job (from Figure A.1) on the farequote dataset that has an anomaly on the date of February 9th:

Figure A.10 – An analysis on the farequote dataset with an anomaly we'd like to ignore

Now, let's explore the ways we can ignore the anomaly on February 9th using different situations.

Ignoring an upcoming (known) window of time

Two methods can be used to ignore an upcoming window of...

Using custom rules and filters to your advantage

While the anomaly detection jobs are incredibly useful, they are also agnostic to the domain and to the relevance of the raw data. In other words, the unsupervised machine learning algorithms do not know that a tenfold increase in CPU utilization (from 1% to 10%, for example) may not be that interesting to the proper operation of an application even though it may be statistically anomalous/unlikely in the scenario. Likewise, the anomaly detection jobs treat every entity analyzed equally, but the user might want to disavow results for a certain IP address or user ID, since the user knows that anomalies found for these entities are not desired or useful. The usage of custom rules and filters allows the user to inject domain knowledge into the anomaly detection job configuration, thereby having a fair amount of control as to what gets deemed or marked anomalous – or even if entities get considered part of the modeling process in...

Anomaly detection job throughput considerations

Elastic ML is awesome and is no doubt very fast and scalable, but there will still be a practical upper bound of events/second processed to any anomaly detection job, depending on a couple of different factors:

• The speed at which data can be delivered to the algorithms (that is, query performance)
• The speed at which the algorithms can chew through the data, given the desired analysis

For the latter, much of the performance is based upon the following:

• The function(s) chosen for the analysis, that is, count is faster than lat_long
• The bucket_span value chosen (longer bucket spans are faster than smaller bucket spans because more buckets analyzed per unit of time compound the per-bucket processing overhead, which is writing results and so on)

However, if you have a defined analysis set up and can't change it for other reasons, then there's not that much you can do unless you get creative and...

Avoiding the over-engineering of a use case

I once worked with a user where we discussed different use cases for anomaly detection. In particular, this customer was building a hosted security operations center as part of their managed security service provider (MSSP) business, so they were keen to think about use cases in which ML could help.

A high-level theme to their use cases was to look at a user's behavior and find unexpected behavior. One example that was discussed was login activity from unusual/rare locations such as Bob just logged in from Ukraine, but he doesn't normally log in from there.

In the process of thinking the implementation through, there was talk of them having multiple clients, each of which had multiple users. Therefore, they were thinking of ways to split/partition the data so that they could execute rare by country for each and every user of every client.

I asked them to take a step back and said, "Is it worthy of an anomaly if anyone...

Using anomaly detection on runtime fields

In some cases, it might be necessary to analyze the value of a field that doesn't exist in the index mappings but can be calculated dynamically from other field values. This capability to dynamically define field values has existed for quite some time in Elasticsearch as script fields, but starting in v7.11, script fields are replaced by an updated concept known as runtime fields. In short, runtime fields are treated like first-class citizens in the Elasticsearch mapping (if defined there) and will eventually allow the user to promote a runtime field into an indexed field.

Users can define runtime fields in the mapping or only in the search request. It is good to note that at the time of writing, there is no support for definitions of runtime fields in the data feed of an anomaly detection job. However, if the runtime fields are defined in the mappings, then the anomaly detection job can leverage them seamlessly.

Note

For more...

Summary

Elastic ML is a powerful, flexible, yet easy-to-use feature that gives the power of data science to non-data scientists so that they can gain insight into massive amounts of data. Throughout this entire book, there are many different ways in which users can take advantage of technology to solve real-world challenges in IT. We hope that you will take the knowledge that you have gained in this book and implement some great use cases of your own. Don't worry about solving all possible problems on day 1 – start small, get some tangible wins, and grow your usage as you gain more confidence. Success will breed success!

Why subscribe?

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
• Improve your learning with Skill Plans built especially for you
• Get a free eBook or video every month
• Fully searchable for easy access to vital information
• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [e-mail] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.