The theory of attention within neural networks

In the previous chapter, in our sequence-to-sequence model for sentence translation (with no attention implemented), we used both encoders and decoders. The encoder obtained a hidden state from the input sentence, which was a representation of our sentence. The decoder then used this hidden state to perform the translation steps. A basic graphical illustration of this is as follows:

Figure 8.1 – Graphical representation of sequence-to-sequence models

However, decoding over the entirety of the hidden state is not necessarily the most efficient way of using this task. This is because the hidden state represents the entirety of the input sentence; however, in some tasks (such as predicting the next word in a sentence), we do not need to consider the entirety of the input sentence, just the parts that are relevant to the prediction we are trying to make. We can show that by using attention within our sequence...