Chapter 12: Performance Metrics

No model of a real-world phenomenon is perfect. There are countless statistical assumptions made about the underlying data, there is noise in the measurements, and there are unknown and unmodeled contributing factors to the output. But even though it is not perfect, a good model is still informative and valuable. So, how do you know whether you have such a good model? How can you be sure your predictions for the future can be trusted? Cross-validation got us part of the way there, by providing a technique to compare unbiased predictions to actual values. This chapter is all about how to compare different models.

Prophet features several different metrics that are used for comparing your actual values with your predicted values, so you can quantify the performance of your model. This tells you how good your model actually is and whether you can trust the predictions, and helps you compare the performance of different models so you can choose which...

Technical requirements

The data files and code for examples in this chapter can be found at https://github.com/PacktPublishing/Forecasting-Time-Series-Data-with-Facebook-Prophet.

Understanding Prophet's metrics

Prophet's diagnostics package provides six different metrics you can use to evaluate your model. Those metrics are mean squared error, root mean squared error, mean absolute error, mean absolute percent error, median absolute percent error, and coverage. We'll discuss each of these in turn.

Mean squared error

Mean squared error (MSE) is the sum of the squared difference between each predicted value and the actual value, as can be seen in the following equation:

Figure 12.1 – Mean squared error

The number of samples is represented in the preceding equation with n, while y is an actual value and ŷ a forecasted value.

MSE may be the most used performance metric, but it does have its downside. Because it is not scaled to the data, its value is not easy to interpret – the unit of MSE is the square of your y unit. It is also sensitive to outliers, although this may be either desirable or...

Creating the Prophet performance metrics DataFrame

Now that you've learned what the different options are for performance metrics in Prophet, let's start coding and see how to access these. We'll use the same online retail sales data we used in Chapter 11, Cross-Validation. Along with our usual imports, we are going to add the performance_metrics function from Prophet's diagnostics package and the plot_cross_validation_metric function from the plot package:

import pandas as pd
import matplotlib.pyplot as plt
from fbprophet import Prophet
from fbprophet.plot import add_changepoints_to_plot
from fbprophet.diagnostics import cross_validation
from fbprophet.diagnostics import performance_metrics
from fbprophet.plot import plot_cross_validation_metric

Next, let's load the data, create our forecast, and plot the results:

df = pd.read_csv('online_retail.csv')
df.columns = ['ds', 'y']
model = Prophet(yearly_seasonality=4)
model...

Handling irregular cut-offs

We'll be using a new dataset for this example. The World Food Programme (WFP) is the branch of the United Nations focused on hunger and food security. One of the greatest contributing factors to food security issues in developing countries that the WFP tracks is rainfall amounts because it can affect agricultural production. Thus, predicting rainfall is of critical importance in planning aid delivery.

This data represents the rainfall received over 30 years in one of the regions the WFP monitors. What makes this dataset unique is that the WFP recorded the amount of rain that accumulated at three times per month, on the 1st, the 11th, and the 21st. The accumulation from the 1st to the 11th is a 10-day period. It's the same with the 11th to the 21st. But the period from the 21st of one month to the 1st of the next varies depending upon the month. In a normal February, it will be 8 days. In a leap year, 9 days. Months of 30 and 31 days will see...

Tuning hyperparameters with grid search

For the final section in this chapter, we'll look at grid search and work through an example, continuing with this rainfall data. If you're not familiar with the concept of grid search, it's a way to exhaustively check all reasonable combinations of hyperparameters against a performance indicator and choose the best combination to train your final model. With Prophet, you might decide to select the following hyperparameters and values:

Figure 12.13 – Prophet grid search parameters

With these parameters, a grid search will iterate through each unique combination, use cross-validation to calculate and save a performance metric, and then output the set of parameter values which resulted in the best performance.

Prophet does not have a grid search method the way, for example, sklearn does. One is easy enough to build yourself in Python though, so let's see how to set it up. The first step is...

Summary

In this chapter, you learned how to use Prophet's performance metrics to extend the usefulness of cross-validation. You learned about the six metrics Prophet has out of the box, namely mean squared error, root mean squared error, mean absolute error, mean absolute percent error, median absolute percent error, and coverage. You learned many of the advantages and disadvantages of these metrics, and situations where you may want to use or avoid any one of them.

Next, you learned how to create Prophet's performance metrics DataFrame and use it to create a plot of your preferred cross-validation metric so as to be able to evaluate the performance of your model on unseen data across a range of forecast horizons. You then used this plot with the World Food Programme's rainfall data to see a situation where Prophet's automatic cut-off date selection is not ideal, and how to create custom cut-off dates.

Finally, you brought all of this together in an exhaustive...