Building RNNs

RNNs consist of recurrent layers. While they are similar in many ways to the fully connected layers within a standard feed forward neural network, these recurrent layers consist of a hidden state that is updated at each step of the sequential input. This means that for any given sequence, the model is initialized with a hidden state, often represented as a one-dimensional vector. The first step of our sequence is then fed into our model and the hidden state is updated depending on some learned parameters. The second word is then fed into the network and the hidden state is updated again depending on some other learned parameters. These steps are repeated until the whole sequence has been processed and we are left with the final hidden state. This computation loop, with the hidden state carried over from the previous computation and updated, is why we refer to these networks as recurrent. This final hidden state is then connected to a further fully connected layer and...