How recurrent neural nets work
RNNs assume that the input data has been generated as a sequence such that previous data points impact the current observation and are relevant for predicting subsequent values. Thus, they allow more complex input-output relationships than FFNNs and CNNs, which are designed to map one input vector to one output vector using a given number of computational steps. RNNs, in contrast, can model data for tasks where the input, the output, or both, are best represented as a sequence of vectors. For a good overview, refer to Chapter 10 in Goodfellow, Bengio, and Courville (2016).
The diagram in Figure 19.1, inspired by Andrew Karpathy's 2015 blog post The Unreasonable Effectiveness of Recurrent Neural Networks (see GitHub for a link), illustrates mappings from input to output vectors using nonlinear transformations carried out by one or more neural network layers:
Figure 19.1: Various types of sequence-to-sequence models
The left panel...
