Chapter 1, From Bag-of-Words to the Transformers, provides a brief introduction to the history of NLP, providing a comparison between traditional methods, deep learning models such as CNNs, RNNs, and LSTMs, and transformer models.
Chapter 2, A Hands-On Introduction to the Subject, takes a deeper look at how a transformer model can be used. Tokenizers and models such as BERT will be described with hands-on examples.
Chapter 3, Autoencoding Language Models, is where you will gain knowledge about how to train autoencoding language models on any given language from scratch. This training will include pretraining and the task-specific training of models.
Chapter 4, Autoregressive and Other Language Models, explores the theoretical details of autoregressive language models and teaches you about pretraining them on their own corpus. You will learn how to pretrain any language model such as GPT-2 on their own text and use the model in various tasks such as language generation.
Chapter 5, Fine-Tuning Language Models for Text Classification, is where you will learn how to configure a pre-trained model for text classification and how to fine-tune it for any text classification downstream task, such as sentiment analysis or multi-class classification.
Chapter 6, Fine-Tuning Language Models for Token Classification, teaches you how to fine-tune language models for token classification tasks such as NER, POS tagging, and question-answering.
Chapter 7, Text Representation, is where you will learn about text representation techniques and how to efficiently utilize the transformer architecture, especially for unsupervised tasks such as clustering, semantic search, and topic modeling.
Chapter 8, Working with Efficient Transformers, shows you how to make efficient models out of trained models by using distillation, pruning, and quantization. Then, you will gain knowledge about efficient sparse transformers, such as Linformer and BigBird, and how to work with them.
Chapter 9, Cross-Lingual and Multilingual Language Modeling, is where you will learn about multilingual and cross-lingual language model pretraining and the difference between monolingual and multilingual pretraining. Causal language modeling and translation language modeling are the other topics covered in the chapter.
Chapter 10, Serving Transformer Models, will detail how to serve transformer-based NLP solutions in environments where CPU/GPU is available. Using TensorFlow Extended (TFX) for machine learning deployment will be described here also.
Chapter 11, Attention Visualization and Experiment Tracking, will cover two different technical concepts: attention visualization and experiment tracking. We will practice them using sophisticated tools such as exBERT and BertViz.
