From one token to an AI revolution
Yes, the title is correct, as you will see in this section. One token produced an AI revolution and has opened the door to AI in every domain and application.
ChatGPT with GPT-4, PaLM 2, and other LLMs have a unique way of producing text.
In LLMs, a token is a minimal word part. The token is where a Large Language Model starts and ends.
For example, the word including could become: includ + ing, representing two tokens. GPT models predict tokens based on the hundreds of billions of tokens in its training dataset. Examine the graph in Figure 1.9 of an OpenAI GPT model that is making an inference to produce a token:
Figure 1.9: GPT inference graph built in Python with NetworkX
It may come as a surprise, but the only parts of this figure controlled by the model are Model and Output Generation!, which produce raw logits. All the rest is in the pipeline.
To understand the pipeline, we will first go through the description...
