Evaluating deep neural networks for forecasting

Evaluating the performance of forecasting models is essential to understand how well they generalize to unseen data. Popular metrics include the Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE), Mean Absolute Scaled Error (MASE), and Symmetric Mean Absolute Percentage Error (SMAPE), among others. We will implement these metrics in Python and show you how they can be applied to evaluate our model’s performance.

Getting ready

We need predictions from our trained model and the corresponding ground truth values to calculate these metrics. Therefore, we must run our model on the test set first to obtain the predictions.

To simplify the implementation, we will use the scikit-learn and sktime libraries since they have useful classes and methods to help us with this task. Since we have not installed sktime yet, let’s run the following command:

pip install sktime

Now, it is time to import the classes...