Google Colab Free with a CPU
It is nearly impossible to fine-tune or train a transformer model with millions or billions of parameters on a CPU. CPUs are mostly sequential. Transformer models are designed for parallel processing.
In the Runtime menu and Change Runtime Type submenu, you can select a hardware accelerator: None (CPU), GPU, or TPU.
This test was run with None (CPU), as shown in Figure II.2:
Figure II.2: Selecting a hardware accelerator
When the notebook reaches the training loop, it slows down right from the start:
Figure II.3: Training loop
After 15 minutes, nothing has really happened.
CPUs are not designed for parallel processing. Transformer models are designed for parallel processing, so part from toy models, they require GPUs.
Google Colab Free with a GPU
Let’s go back to the notebook settings to select a GPU.
Figure II.4 Selecting a GPU
At the time of writing, I tested Google Colab, and an NVIDIA...
