Attention has proven to be so effective in machine translation that it has been expanded into natural language processing, statistical learning, speech understanding, object detection and recognition, image captioning, and visual question answering.
The purpose of attention is to estimate how correlated (connected) two or more elements are to one another.
However, there isn't just one kind of attention. There are many types, such as the following:
- Self-attention: Captures the relationship between different positions of a sequence of inputs
- Global or soft attention: Focuses on the entire sequence of inputs
- Local or hard attention: Focuses on only part of the sequence of inputs
Let's take a look at these in more detail.
