Ensembling and Stacking

In the previous chapter, we looked at a few machine learning algorithms and used them to generate forecasts on the London Smart Meters dataset. Now that we have multiple forecasts for all the households in the dataset, how do we come up with a single forecast by choosing or combining these different forecasts? That is what we will be doing in this chapter – we will learn how to leverage combinatorial and mathematical optimization to come up with a single forecast.

In this chapter, we will cover the following topics:

• Strategies for combining forecasts
• Stacking or blending

Technical requirements

You will need to set up the Anaconda environment following the instructions in the Preface of the book to get a working environment with all the packages and datasets required for the code in this book.

You need to run the following notebooks for this chapter:

• 02 - Preprocessing London Smart Meter Dataset.ipynb in Chapter02
• 01-Setting up Experiment Harness.ipynb in Chapter04
• 02-Baseline Forecasts using darts.ipynb in Chapter04
• 01-Feature Engineering.ipynb in Chapter06
• 02-Dealing with Non-Stationarity.ipynb in Chapter07
• 02a-Dealing with Non-Stationarity-Train+Val.ipynb in Chapter07
• 00-Single Step Backtesting Baselines.ipynb in Chapter08
• 01-Forecasting with ML.ipynb in Chapter08
• 01a-Forecasting with ML for Test Dataset.ipynb in Chapter08
• 02-Forecasting with Target Transformation.ipynb in Chapter08
• 02a-Forecasting with Target Transformation(Test).ipynb in Chapter08

The code for this chapter can be found at...

Combining forecasts

We have generated forecasts by using many techniques – some univariate, some machine learning, and so on. But at the end of the day, we would need a single forecast, and that means choosing a forecast or combining a variety. The most straightforward option is to choose the algorithm that does the best in the validation dataset, which in our case is LightGBM. We can think of this selection as another function that takes the forecasts that we generated as inputs and combines them into a final forecast. Mathematically, this can be represented as follows:

Here, is the function that combines N forecasts. We can use the function to choose the best-performing model in the validation dataset. However, this function can be as complex as it wants to be, and choosing the right function while balancing bias and variance is a must.

Notebook alert

To follow along with the code, use the 01-Forecast Combinations.ipynb notebook in the chapter09...

Stacking or blending

We started this chapter by talking about machine learning algorithms, which learn a function from a set of inputs and outputs. While using those machine learning algorithms, we learned about the functions that forecast our time series, which we'll call base forecasts now. Why not use the same machine learning paradigm to learn this new function, , that we are trying to learn as well?

This is exactly what we do in stacking (often called stacked generalization), where we train another learning algorithm on the predictions of some base learners to combine these predictions. This second-level model is often called a stacked model or a meta model. And typically, this meta model performs equal to or better than the base learners.

Although the idea originated with Wolpert in 1992, Leo Breiman formalized this idea in the way it is used now in his 1996 paper titled Stacked Regressions. And in 2007, Mark J. Van der Laan et al. established the theoretical underpinnings...

Summary

Continuing with the streak of practical lessons in the previous chapter, we completed yet another hands-on lesson. In this chapter, we generated forecasts from different machine learning models from the previous chapter. We learned how to combine these different forecasts into a single forecast that performs better than any single model. Then, we explored concepts such as combinatorial optimization and stacking to achieve state-of-the-art results.

In the next chapter, we will start talking about global models of forecasting and explore strategies, feature engineering, and so on to enable such modeling.

References

The following references were provided in this chapter:

1. David S. Johnson, Cecilia R. Aragon, Lyle A. McGeoch, and Catherine Schevon (1989), Optimization by Simulated Annealing: An Experimental Evaluation; Part I, Graph Partitioning. Operations Research, 1989, vol. 37, issue 6, 865-892 – http://dx.doi.org/10.1287/opre.37.6.865
2. L. Breiman (1996), Stacked regressions. Mach Learn 24, 49–64 – https://doi.org/10.1007/BF00117832
3. Mark J. van der Laan; Eric C.Polley; and Alan E.Hubbard (2007), Super Learner. U.C. Berkeley Division of Biostatistics Working Paper Series. Working Paper 222: https://biostats.bepress.com/ucbbiostat/paper222

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• A Kaggler’s Guide to Model Stacking in Practice, by Ha Nguyen: https://datasciblog.github.io/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/
• Kai Ming Ting and Ian H. Witten (1997), Stacked Generalization: when does it work?: https://www.ijcai.org/Proceedings/97-2/Papers/011.pdf
• Pablo Montero-Manso, George Athanasopoulos, Rob J. Hyndman, Thiyanga S. Talagala (2020), FFORMA: Feature-based forecast model averaging. International Journal of Forecasting, Volume 36, Issue 1: https://robjhyndman.com/papers/fforma.pdf
• Peiyi Zhang, et al. (2021), Self-supervised learning for fast and scalable time-series hyper-parameter tuning: https://www.ijcai.org/Proceedings/97-2/Papers/011.pdf
• Local versus Global Optima: https://www.mathworks.com/help/optim/ug/local-vs-global-optima.html