Advanced Deep Learning Architectures for Time Series Forecasting

In previous chapters, we’ve learned how to create forecasting models using different types of neural networks but, so far, we’ve worked with basic architectures such as feedforward neural networks or LSTMs. This chapter describes how to build forecasting models with state-of-the-art approaches such as DeepAR or Temporal Fusion Transformers. These have been developed by tech giants such as Google and Amazon and are available in different Python libraries. These advanced deep learning architectures are designed to tackle different types of forecasting problems.

We’ll cover the following recipes:

• Interpretable forecasting with N-BEATS
• Optimizing the learning rate with PyTorch Forecasting
• Getting started with GluonTS
• Training a DeepAR model with GluonTS
• Training a Transformer with NeuralForecast
• Training a Temporal Fusion Transformer with GluonTS
• Training an Informer...

Technical requirements

This chapter requires the following Python libraries:

• numpy (1.23.5)
• pandas (1.5.3)
• scikit-learn (1.2.1)
• sktime (0.24.0)
• torch (2.0.1)
• pytorch-forecasting (1.0.0)
• pytorch-lightning (2.1.0)
• gluonts (0.13.5)
• neuralforecast (1.6.0)

You can install these libraries in one go using pip:

pip install -U pandas numpy scikit-learn sktime torch pytorch-forecasting pytorch-lightning gluonts neuralforecast

The code for this chapter can be found at the following GitHub URL: https://github.com/PacktPublishing/Deep-Learning-for-Time-Series-Data-Cookbook.

Interpretable forecasting with N-BEATS

This recipe introduces Neural Basis Expansion Analysis for Interpretable Time Series Forecasting (N-BEATS), a deep learning method for forecasting problems. We’ll show you how to train N-BEATS using PyTorch Forecasting and interpret its output.

Getting ready

N-BEATS is particularly designed for problems involving several univariate time series. So, we’ll use the dataset introduced in the previous chapter (check, for example, the Preparing multiple time series for a global model recipe):

import numpy as np
import pandas as pd
from gluonts.dataset.repository.datasets import get_dataset
from pytorch_forecasting import TimeSeriesDataSet
import lightning.pytorch as pl
from sklearn.model_selection import train_test_split
dataset = get_dataset('nn5_daily_without_missing', regenerate=False)
N_LAGS = 7
HORIZON = 7
datamodule = GlobalDataModule(data=dataset,
    n_lags=N_LAGS,
    ...

Optimizing the learning rate with PyTorch Forecasting

In this recipe, we show how to optimize the learning rate of a model based on PyTorch Forecasting.

Getting ready

The learning rate is a cornerstone parameter of all deep learning methods. As the name implies, it controls how quickly the learning process of the network is. In this recipe, we’ll use the same setup as the previous recipe:

datamodule = GlobalDataModule(data=dataset,
    n_lags=N_LAGS,
    horizon=HORIZON,
    batch_size=32,
    test_size=0.2)
datamodule.setup()

We’ll also use N-BEATS as an example. However, the process is identical for all models based on PyTorch Forecasting.

How to do it…

The optimization of the learning rate can be carried out using the Tuner class from PyTorch Lightning. Here is an example with N-BEATS:

from lightning.pytorch.tuner import Tuner
import lightning.pytorch as pl
from...

Getting started with GluonTS

GluonTS is a flexible and extensible toolkit for probabilistic time series modeling using PyTorch. The toolkit provides state-of-the-art deep learning architectures specifically designed for time series tasks and an array of utilities for time series data processing, model evaluation, and experimentation.

The main objective of this section is to introduce the essential components of the gluonts library, emphasizing its core functionalities, adaptability, and user-friendliness.

Getting ready

To begin our journey, ensure that gluonts is installed as well as its backend dependency, pytorch:

pip install gluonts pytorch

With the installations complete, we can now dive into the capabilities of gluonts.

How to do it…

We start by accessing a sample dataset provided by the library:

from gluonts.dataset.repository.datasets import get_dataset
dataset = get_dataset("nn5_daily_without_missing", regenerate=False)

This will load...

Training a DeepAR model with GluonTS

DeepAR is a state-of-the-art forecasting method that utilizes autoregressive recurrent networks to predict future values of time series data. Amazon introduced it; it was designed for forecasting tasks that can benefit from longer horizons, such as demand forecasting. The method is particularly powerful when there’s a need to generate forecasts for multiple related time series.

Getting ready

We’ll use the same dataset as in the previous recipe:

from gluonts.dataset.repository.datasets import get_dataset
dataset = get_dataset("nn5_daily_without_missing", regenerate=False)

Now, let’s see how to build a DeepAR model with this data.

How to do it…

We start by formatting the data for training:

1. Let’s do this by using the ListDataset data structure:

   from gluonts.dataset.common import ListDataset
   from gluonts.dataset.common import FieldName
   train_ds = ListDataset(
       ...

Training a Transformer model with NeuralForecast

Now, we turn our attention to Transformer architectures that have been driving recent advances in various fields of artificial intelligence. In this recipe, we will show you how to train a vanilla Transformer using the NeuralForecast Python library.

Getting ready

Transformers have become a dominant architecture in the deep learning community, especially for natural language processing (NLP) tasks. Transformers have been adopted for various tasks beyond NLP, including time series forecasting.

Unlike traditional models that analyze time series data point by point in sequence, Transformers evaluate all time steps simultaneously. This approach is similar to observing an entire timeline at once, determining the significance of each moment in relation to others for a specific point in time.

At the core of the Transformer architecture is the attention mechanism. This mechanism calculates a weighted sum of input values, or values...

Training a Temporal Fusion Transformer with GluonTS

The TFT is an attention-based architecture developed at Google. It has recurrent layers to learn temporal relationships at different scales combined with self-attention layers for interpretability. TFTs also use variable selection networks for feature selection, gating layers to suppress unnecessary components, and quantile loss as their loss function to produce forecasting intervals.

In this section, we delve into training and performing inference with a TFT model using the GluonTS framework.

Getting ready

Ensure you have the GluonTS library and PyTorch backend installed in your environment. We’ll use the nn5_daily_without_missing dataset from the GluonTS repository as a working example:

from gluonts.dataset.common import ListDataset, FieldName
from gluonts.dataset.repository.datasets import get_dataset
dataset = get_dataset("nn5_daily_without_missing", regenerate=False)
train_ds = ListDataset(
 ...

Training an Informer model with NeuralForecast

In this recipe, we’ll explore the neuralforecast Python library to train an Informer model, another Transformer-based deep learning approach for forecasting.

Getting ready

Informer is a Transformer method tailored for long-term forecasting – that is, predicting with a large forecasting horizon. The main difference relative to a vanilla Transformer is that Informer provides an improved self-attention mechanism, which significantly reduces the computational requirements to run the model and generate long-sequence predictions.

In this recipe, we’ll show you how to train Informer using neuralforecast. We’ll use the same dataset as in the previous recipes:

from gluonts.dataset.repository.datasets import get_dataset
dataset = get_dataset('nn5_daily_without_missing')

How to do it…

This time, instead of creating DataModule to handle the data preprocessing, we’ll use the typical...

Comparing different Transformers with NeuralForecast

NeuralForecast contains several deep learning methods that you can use to tackle time series problems. In this recipe, we’ll walk you through the process of comparing different Transformer-based models using neuralforecast.

Getting ready

We’ll use the same dataset as in the previous recipe (the df object). We set the validation and test size to 10% of the data size each:

val_size = int(.1 * n_time)
test_size = int(.1 * n_time)

Now, let’s see how to compare different models using neuralforecast.

How to do it…

We start by defining the models we want to compare. In this case, we’ll compare an Informer model with a vanilla Transformer, which we set up as follows:

from neuralforecast.models import Informer, VanillaTransformer
models = [
    Informer(h=HORIZON,
        input_size=N_LAGS,
      ...