Chapter 4: Forecasting

Forecasting is a natural extension of the time series modeling of Elastic ML. Since very expressive models are built behind the scenes and describe how data has behaved historically, it is therefore possible to project that information forward in time and predict how something should behave at a future time.

We will spend time learning the concepts behind forecasting, as well as stepping through some practical examples.

Specifically, this chapter will cover the following topics:

• Contrasting forecasting with prophesying
• Forecasting use cases
• Forecasting theory of operation
• Single time series forecasting
• Looking at forecasting results
• Multiple time series forecasting

Technical requirements

The information and examples demonstrated in this chapter are relevant as of v7.11 of the Elastic Stack.

Contrasting forecasting with prophesying

Past performance is not indicative of future results. This disclaimer is used by financial companies when they reference the performance of products such as mutual funds. But this disclaimer is a bit of an odd contradiction, because the past is all that we have to work with. If the companies that comprise the mutual fund have had consistently positive quarterly results for the last eight quarters straight, does that guarantee that they will also have a positive set of results for the next eight quarters and that their public valuation will continue to rise? Probability could be on the side of that being the case, but that might not be the whole story. And, before we get too wishful in thinking that Elastic ML’s ability to forecast is our key to making a fortune in the stock market, we should be realistic about one key caveat—there are always uncontrollable factors.

The reason financial companies use the preceding disclaimer...

Forecasting use cases

In the context of Elastic ML, there are really just two—somewhat similar—use cases in which someone would use forecasting. These are outlined here:

• Value-focused: This is where you extrapolate a time series into the future to understand a probable future value. This would be akin to answering questions such as: “How many widgets will I sell per day 2 months from now?”
• Time-focused: This is where you understand the likely time at which an expected value is to be reached. This would answer questions similar to: “Do I expect to reach 80% utilization in the next week?”

The differences between these two use cases might not just be how a question is asked (how the data is searched), but also how you interpret the output. However, before we delve into a few examples of how to use the forecasting feature, let’s take a little time to discuss how it works logistically.

Forecasting theory of operation

The first thing to realize is that the act of invoking a forecast on data is that it is an extension of an existing Anomaly Detection job. In other words, you need to have an Anomaly Detection job configured, and that job needs to have analyzed historical data before you can forecast on that data. This is because the forecasting process uses the models that are created by the Anomaly Detection job. To forecast the data, you need to follow the same steps that were used to create an Anomaly Detection job as described in other chapters. If anomalies were generated by the execution of that job, you can disregard them if your only purpose is to execute forecasting. Once the job has learned on some historical data, the model or models (if the job is configured to analyze data from more than one time series) associated with that job are current and up to date, as represented in the following diagram:

Figure 4.1 – A symbolic representation...

Single time series forecasting

To illustrate the procedure of forecasting, we will start with a dataset that is a single time series. While this dataset is generic, you could imagine that it could represent a system performance metric, the number of transactions processed by a system, or even sales revenue data. The important aspect of this dataset is that it contains several distinct time-based trends—a daily trend, a weekly trend, and an overall increasing trend. Elastic ML will discover all three trends and will effectively predict those into the future. It is good to note that the dataset also contains some anomalies, but (of course) future anomalies cannot be predicted as they are surprise events by definition. Since our discussion here is purely focused on forecasting, we will ignore the existence of any anomalies found in our dataset while building the models for forecasting.

With that said, let’s jump into an example by using the forecast_example dataset from...

Looking at forecast results

Now that we have run a forecast, we can look in more depth at the results that are generated by the forecasting process. We can view the results of a previously created forecast at any time in the UI via one of two methods. The first way is to click the Forecast button in Single Metric Viewer to reveal a list of previous forecasts, like so:

Figure 4.20 – Viewing previously created forecasts from Single Metric Viewer

Alternatively, you can view them in the Job Management page under the Forecasts tab for that job, as illustrated in the following screenshot:

Figure 4.21 – Viewing previously created forecasts from the Job Management page

Note

Forecast results built in Kibana have a default lifespan of 14 days. After that, the forecast results are deleted permanently. If a different expiration duration is desired, then the forecast will have to be invoked via the _forecast API endpoint, which...

Multiple time series forecasting

To invoke forecasting on multiple time series, you simply just need an ML job that is modeling multiple time series. Let’s assume that we have an ML job that has analyzed web requests per country. In fact, using the built-in sample web logs (kibana_sample_data_logs) we used in Chapter 3, Anomaly Detection, we could easily create a multi-metric job that counts events, split on the source country code of the request (the field is called geo.src), as illustrated in the following screenshot:

Figure 4.26 – Creating a multi-metric job for forecasting

There are 183 unique source countries in this dataset. After creating and running this Anomaly Detection job in order to build baseline models for all 183 countries, we are now in a position to invoke a forecast. If we approach the invocation of a forecast in the same way as we did before (via Single Metric Viewer), we might erroneously think that a forecast will only be...

Summary

Elastic ML has an additional feature over and above anomaly detection: the ability to take and extrapolate time series models into the future for forecasting purposes. With use cases that include advanced breach detection and capacity planning, this feature alleviates the human burden of manually charting, tracking, and predicting where things are going in the future, based upon how they have behaved in the past.

In the next chapter, we’ll go deeper into the results that anomaly detection and forecasting give us, and we will set up a better understanding of how to leverage those results for dashboards and proactive alerts.