How could this deceivingly simple O(1) time complexity class forever change AI and our everyday lives? How could O(1) explain the profound architectural changes that made ChatGPT so powerful and stunned the world? How can something as simple as O(1) allow systems like ChatGPT to spread to every domain and hundreds of tasks?

The answer to these questions is the only way to find your way in the growing maze of transformer datasets, models, and applications is to focus on the underlying concepts of thousands of assets. Those concepts will take you to the core of the functionality you need for your projects.

This section will provide a significant answer to those questions before we move on to see how one token (a minimal piece of a word) started an AI revolution that is raging around the world, triggering automation never seen before.

We need to get to the bottom of the chaos and disruption generated by transformers...

Yes, the title is correct, as you will see in this section. One token produced an AI revolution and has opened the door to AI in every domain and application.

ChatGPT with GPT-4, PaLM 2, and other LLMs have a unique way of producing text.

In LLMs, a token is a minimal word part. The token is where a Large Language Model starts and ends.

For example, the word including could become: includ + ing, representing two tokens. GPT models predict tokens based on the hundreds of billions of tokens in its training dataset. Examine the graph in Figure 1.9 of an OpenAI GPT model that is making an inference to produce a token:

Figure 1.9: GPT inference graph built in Python with NetworkX

It may come as a surprise, but the only parts of this figure controlled by the model are Model and Output Generation!, which produce raw logits. All the rest is in the pipeline.

To understand the pipeline, we will first go through the description...

Advanced large multipurpose transformer models represent such a paradigm change that they require a new name to describe them: Foundation Models. Accordingly, Stanford University created the Center for Research on Foundation Models (CRFM). In August 2021, the CRFM published a two-hundred-page paper (see the References section) written by over one hundred scientists and professionals: On the Opportunities and Risks of Foundation Models.

Foundation Models were not created by academia but by the big tech industry. Google invented the transformer model, leading to Google BERT, LaMBDA, PaLM 2, and more. Microsoft partnered with OpenAI to produce ChatGPT with GPT-4, and soon more.

Big tech had to find a better model to face the exponential increase of petabytes of data flowing into their data centers. Transformers were thus born out of necessity.

Let’s consider the evolution of LLMs to understand the need for industrialized AI models.

Transformers...

Transformer-driven AI is connecting everything to everything, everywhere. Machines communicate directly with other machines. AI-driven IoT signals trigger automated decisions without human intervention. NLP algorithms send automated reports, summaries, emails, advertisements, and more.

AI specialists must adapt to this new era of increasingly automated tasks, including transformer model implementations. AI professionals will have new functions. If we list transformer NLP tasks that an AI specialist will have to do, from top to bottom, it appears that some high-level tasks require little to no development from an AI specialist. An AI specialist can be an AI guru, providing design ideas, explanations, and implementations.

The pragmatic definition of what a transformer represents for an AI specialist will vary with the ecosystem.

Let’s go through a few examples:

• API: The OpenAI API does not require an AI developer. A web designer...

We are now well into the industrialization era of AI. Microsoft Azure, Google Cloud, Amazon Web Services (AWS), and IBM, among others, provide AI services that no developer or team of developers could hope to outperform. Tech giants have million-dollar supercomputers with massive datasets to train transformer models and AI models in general.

Big tech giants have many corporate customers that already use their cloud services. As a result, adding a transformer API to an existing cloud architecture requires less effort than any other solution.

A small company or even an individual can access the most powerful transformer models through an API with practically no investment in development. An intern can implement the API in a few days. There is no need to be an engineer or have a Ph.D. for such a simple implementation.

For example, the OpenAI platform now has a Software as a Service (SaaS) API for some of the most effective...

Transformers forced AI to make profound evolutions. Foundation Models, including their Generative AI abilities, are built on top of the digital revolution connecting everything to everything with underlying processes everywhere. Automated processes are replacing human decisions in critical areas, including NLP.

RNNs slowed the progression of automated NLP tasks required in a fast-moving world. Transformers filled the gap. A corporation needs summarization, translation, and a wide range of NLP tools to meet the challenges of the growing volume of incoming information.

Transformers have thus spurred an age of AI industrialization. We first saw how the O(1) time complexity of the attention layers and their computational time complexity, O(n²*d), shook the world of AI.

We saw how the one-token flexibility of transformer models pervaded every domain of our everyday lives!

Platforms such as Hugging Face, Google Cloud, OpenAI, and Microsoft Azure provide NLP tasks...

1. ChatGPT is a game-changer. (True/False)
2. ChatGPT can replace all AI algorithms. (True/False)
3. AI developers will sometimes have no AI development to do. (True/False)
4. AI developers might have to implement transformers from scratch. (True/False)
5. It’s not necessary to learn more than one transformer ecosystem, such as Hugging Face. (True/False)
6. A ready-to-use transformer API can satisfy all needs. (True/False)
7. A company will accept the transformer ecosystem a developer knows best. (True/False)
8. Cloud transformers have become mainstream. (True/False)
9. A transformer project can be run on a laptop. (True/False)
10. AI specialists will have to be more flexible. (True/False)

• Bommansani et al., 2021, On the Opportunities and Risks of Foundation Models: https://arxiv.org/abs/2108.07258
• Rishi Bommasani, Dilara Soylu, Thomas I. Liao, Kathleen A. Creel, and Percy Liang, 2023, Ecosystem Graphs: The Social Footprint of Foundation Models: https://arxiv.org/abs/2303.15772
• Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, 2017, Attention is All You Need: https://arxiv.org/abs/1706.03762
• Chen et al., 2021, Evaluating Large Language Models Trained on Code: https://arxiv.org/abs/2107.03374
• Microsoft AI: https://innovation.microsoft.com/en-us/ai-at-scale
• OpenAI: https://openai.com/
• Google AI: https://ai.google/
• Google Trax: https://github.com/google/trax
• AllenNLP: https://allennlp.org/
• Hugging Face: https://huggingface.co/
• Google Cloud TPU: https://cloud.google.com/tpu/docs/intro-to-tpu

• Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock, 2023, GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models: https://arxiv.org/abs/2303.10130
• Jussi Heikkilä, Julius Rissanen, and Timo Ali-Vehmas, 2023, Coopetition, standardization, and general purpose technologies: A framework and an application: https://www.sciencedirect.com/science/article/pii/S0308596122001902
• NVIDIA blog on Foundation Models: https://blogs.nvidia.com/blog/2023/03/13/what-are-foundation-models/
• On Markov chains: https://mathshistory.st-andrews.ac.uk/Biographies/Markov/

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers:

https://www.packt.link/Transformers