Chapter 13: Deep Learning for Time Series Forecasting

If you have been searching the web for topics on data science, artificial intelligence, or machine learning, it is hard to escape headlines on deep learning. Deep learning is a subset of machine learning and excels when dealing with large and complex data, as it can extract complex features with minimal human involvement. Deep learning works well with structured and unstructured data and can be used in supervised, unsupervised, and semi-supervised learning. Several innovations have contributed to its wide adoption, such as the transfer learning technique allowing data scientists to leverage existing pre-trained models, saving a significant development and training time. Transfer learning, the ability to extend a pre-trained model, helped accelerate the adoption of deep learning, which is known to require massive amounts of data, specialized hardware, and longer times to train.

This chapter focuses on using deep learning for time...

Technical requirements

Throughout this chapter, you will be using the same datasets and functions used in Chapter 12, Forecasting Using Supervised Machine Learning. The handle_missing_data and one_step_forecast functions will remain the same.

The Standardize class will be modified slightly to include a split_data method that splits a dataset into train, validation, and test sets. The validation set is used to evaluate the model's performance at each epoch. The following is the updated code for the Standardize class that you will be using throughout this chapter:

1. Start by loading the datasets and preprocessing the time series to be suitable for supervised learning. These are the same steps you followed in Chapter 12, Forecasting Using Supervised Machine Learning:

   Class Standardize:
       def __init__(self, df, split=0.10):
           self.data = df
           self.split = split...

Understanding artificial neural networks

Deep learning utilizes artificial neural networks that consist of connected neurons or nodes. The following diagram represents a shallow neural network (single layer) to highlight the different components:

Figure 13.1 – A single-layer neural network

A network with more than one hidden layer is considered a deep neural network. In Figure 13.1, there are three layers – an input layer, a hidden layer, and an output layer.

The hidden layer represents a layer of connected neurons that perform a mathematical function. In its basic form, a neuron performs a linear function. For example, the first neuron in the hidden layer will perform a simple linear transformation – . Improving how neurons pass information from one layer to another is done by adding an activation function. For example, common activation functions for the hidden layer nodes include Sigmoid, ReLU, or Tanh, which are non-linear functions...

Forecasting with an RNN using Keras

RNNs initially entered the spotlight with Natural Language Processing (NLP), as they were designed for sequential data, where past observations, such as words, have a strong influence on determining the next word in a sentence. This need for the artificial neural network to retain memory (hidden state) inspired the RNN architecture. Similarly, time series data is also sequential, and since past observations influence future observations, it also needs a network with memory. For example, an artificial neural network like the one in Figure 13.1 is considered a Feed-Forward Artificial Neural Network (FFN), as depicted by the arrows pointing from nodes in one layer to the next in one direction; each node has one input and one output. In RNNs, there is a feedback loop where the output of one node or neuron is fed back (the recursive part) as input, allowing the network to learn from a prior time step acting as a memory. Figure 13.3 shows a recurrent cell...

Forecasting with LSTM using Keras

There are a few shortcomings in using RNNs – for example, an RNN's memory is short term and does not do well when persisting a longer-term memory.

In the previous recipe, you trained a small RNN architecture with one hidden layer. In a deep RNN, with multiple hidden layers, the network will suffer from the vanishing gradient problem – that is, during backpropagation, as the weights get adjusted, it will be unable to change the weights of much earlier layers, reducing its ability to learn. Because of this, the output becomes influenced by the closer layers (nodes).

In other words, any memory of earlier layers decays through time, hence the term vanishing. This is an issue if you have a very long sequence – for example, a long paragraph or long sentence – and you want to predict the next word. In time series data, how problematic the lack of long-term memory is will vary, depending on your goal and the data you...

Forecasting with a GRU using Keras

The GRU was proposed as an alternative to the RNN to combat the vanishing gradient problem by introducing the gates concept. As with an LSTM, the gates are used to regulate what and how the data flows. These gates are mathematical functions that act as filters to ensure only significant pieces of information are being retained.

How to do it...

In this recipe, you will continue from the Forecasting with an RNN using Keras recipe. All the time series preprocessing steps and the functions will be the same. The following steps will highlight any necessary changes needed. The energy consumption data is used in the following steps. The Jupyter notebook will include the steps and outputs for other datasets – air passengers and daily temperature:

1. Create another create_model function that is similar to the one you used in the previous recipe. The only difference will involve replacing the RNN layer with the GRU layer. You will use the...

Forecasting with an RNN using PyTorch

In the previous recipes, you used Keras to build different deep learning architectures with minimal changes to code. This is one of the advantages of a high-level API – it allows you to explore and experiment with different architectures very easily.

In this recipe, you will build a simple RNN architecture using PyTorch, a low-level API.

Getting ready

You will be using the functions and steps used to prepare the time series for supervised learning. The one exception is with the features_target_ts function, it will be modified to return a PyTorch Tensor object as opposed to a NumPy ndarray object. In PyTorch, tensor is a data structure similar to NumPy's ndarray object but optimized to work with Graphical Processing Units (GPUs).

You can convert a NumPy ndarray to a PyTorch Tensor object using the torch.from_numpy() method and convert a PyTorch Tensor object to a NumPy ndarray object using the detach.numpy() method:

numpy_array...

Forecasting with LSTM using PyTorch

In this recipe, you will use the same train_model_pt function from the previous Forecasting with an RNN using PyTorch recipe. The function trains the model, captures loss function scores, evaluates the model, makes a forecast using the test set, and finally, produces plots for further evaluation.

You will still need to define a new class for the LSTM model.

How to do it...

The following steps will use the energy consumption data. You can follow the same steps with the other datasets. The Jupyter notebook will include the steps and outputs for other datasets – air passengers  and daily temperature:

1. Create an LSTM class that will inherit from the Module class. The setup will be similar to the RNN class created earlier, but now you have two states (the cell state and the hidden state) and not just one:

   class LSTM(nn.Module):
       def __init__(self, input_size, output_size, n_features, n_layers)...

Forecasting with a GRU using PyTorch

In this recipe, you will use the same train_model_pt function from the previous Forecasting with an RNN using PyTorch recipe. The function trains the model, captures loss function scores, evaluates the model, makes a forecast using the test set, and finally, produces plots for further evaluation.

You will still need to define a new class for the GRU model.

How to do it...

1. Create a GRU class that will inherit from Module class. The setup will be similar to the RNN class, but unlike the LSTM class, you only handle one state (the hidden state):

   class GRU(nn.Module):
       def __init__(self, input_size, output_size, n_features, n_layers):
           super(GRU, self).__init__()
           self.n_layers = n_layers
           self.hidden_dim = n_features
           self.gru...