Pretraining a Generative AI Customer Support Model on X Data

In this section, we will pretrain a Hugging Face RobertaForCausalLM model to be a Generative AI customer support chat agent for X (former Twitter). RoBERTa is an encoder-only model. As such, it is mainly designed to understand and encode inputs. In Chapter 2, Getting Started with the Architecture of the Transformer Model, we saw how the encoder learns to understand and then sends the information to the decoder, generating content. However, in this section, we will use the Hugging Face functionality to adapt a RoBERTa model to run an autoregressive generative AI task. The experiment has limitations, but it shows the inner workings of content generation.The knowledge you acquired in this chapter through building a KantaiBERT from scratch will enable you to enjoy the ride!The generative model and dataset are free, making the exercise particularly interesting. With some work, domain-specific generative AI agents can help companies...

Next steps

You have trained two transformers from scratch. Take some time to imagine what you could do in your personal or corporate environment. You could create a dataset for a specific task and train it from scratch. Many other Hugging Face models are available for training in the BERT family, GPT models, T5, and more!Use your areas of interest or company projects to experiment with the fascinating world of transformer construction kits!Once you have made a model you like, you can share it with the Hugging Face community. Your model will appear on the Hugging Face models page: https://huggingface.co/modelsYou can upload your model in a few steps using the instructions described on this page: https://huggingface.co/transformers/model_sharing.htmlYou can also download models the Hugging Face community has shared to get new ideas for your personal and professional projects.

Summary

In this chapter, we built KantaiBERT, a RoBERTa-like model transformer, from scratch using the building blocks provided by Hugging Face.We first started by loading a customized dataset on a specific topic related to the works of Immanuel Kant. Depending on your goals, you can load an existing dataset or create your own. We saw that using a customized dataset provides insights into how a transformer model thinks. However, this experimental approach has its limits. Training a model beyond educational purposes would take a much larger dataset.The KantaiBERT project was used to train a tokenizer on the kant.txt dataset. The trained merges.txt and vocab.json files were saved. A tokenizer was recreated with our pretrained files. KantaiBERT built the customized dataset and defined a data collator to process the training batches for backpropagation. The trainer was initialized, and we explored the parameters of the RoBERTa model in detail. The model was trained and saved.We saved the...

Questions

1. RoBERTa uses a byte-level byte-pair encoding tokenizer. (True/False)
2. A trained Hugging Face tokenizer produces merges.txt and vocab.json. (True/False)
3. RoBERTa does not use token-type IDs. (True/False)
4. DistilBERT has 6 layers and 12 heads. (True/False)
5. A transformer model with 80 million parameters is enormous. (True/False)
6. We cannot train a tokenizer. (True/False)
7. A BERT-like model has 6 decoder layers. (True/False)
8. Masked Language Modeling (MLM) predicts a word contained in a mask token in a sentence. (True/False)
9. A BERT-like model has no self-attention sublayers. (True/False)
10. Data collators are helpful for backpropagation. (True/False)

References

• Hugging Face Tokenizer documentation: https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=tokenizer
• The Hugging Face reference notebook: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
• The Hugging Face reference blog: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
• More on BERT: https://huggingface.co/transformers/model_doc/bert.html
• More DistilBERT: https://arxiv.org/pdf/1910.01108.pdf
• More on RoBERTa: https://huggingface.co/transformers/model_doc/roberta.html
• Even more on DistilBERT: https://huggingface.co/transformers/model_doc/distilbert.html

Further Reading

• Pretraining of Deep Bidirectional Transformers for Language Understanding, Devlin, et al.(2018): https://arxiv.org/abs/1810.04805
• RoBERTa: A Robustly Optimized BERT Pretraining, Liu et al.(2019) Approach: https://arxiv.org/abs/1907.11692

Join our book’s Discord space

Join the book’s Discord workspace:https://www.packt.link/Transformers

Questions

1. A zero-shot method trains the parameters once. (True/False)
2. Gradient updates are performed when running zero-shot models. (True/False)
3. GPT models only have a decoder stack. (True/False)
4. OpenAI GPT models are not GPTs. (True/False)
5. The diffusion of generative transformer models is very slow in everyday applications. (True/False)
6. GPT-3 models have been useless since GPT-4 was made public. (True/False)
7. ChatGPT models are not completion models. (True/False)
8. Gradio is a transformer model. (True/False)
9. Supercomputers with 285,000 CPUs do not exist. (True/False)
10. Supercomputers with thousands of GPUs are game changers in AI. (True/False)

References

• OpenAI and GPT-3 engines: https://beta.openai.com/docs/engines/engines
• BertViz GitHub Repository by Jesse Vig: https://github.com/jessevig/bertviz
• OpenAI’s supercomputer: https://blogs.microsoft.com/ai/openai-azure-supercomputer/
• Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, 2018, Improving Language Understanding by Generative Pre-Training: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
• Alec Radford et al., 2019, Language Models are Unsupervised Multi-task Learners: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
• Common Crawl data: https://commoncrawl.org/overview/
• GPT-4 Technical Report, OpenAI 2023: https://arxiv.org/pdf/2303.08774.pdf

Further reading

• Alex Wang et.al, 2019, GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding: https://arxiv.org/pdf/1804.07461.pdf
• Alex Wang et al., 2019², SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems: https://w4ngatang.github.io/static/papers/superglue.pdf
• Tom B. Brown et al., 2020, Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165
• Chi Wang et al., 2023, Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference: https://arxiv.org/abs/2303.04673
• Vaswani et al., 2017, Attention Is All You Need: https://arxiv.org/abs/1706.03762

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers:

https://www.packt.link/Transformers