Transfer and Meta Learning

So far in this book, we have studied a variety of neural networks and, as we have seen, each of them has its own strengths and weaknesses with regard to a variety of tasks. We have also learned that deep learning architectures require a large amount of training data because of their size and their large number of trainable parameters. As you can imagine, for a lot of the problems that we want to build models for, it may not be possible to collect enough data, and even if we are able to do so, this would be very difficult and time-consuming—perhaps even costly—to carry out. One way to combat this is to use generative models to create synthetic data (something we encountered in Chapter 8, Regularization) that is generated from a small dataset that we collect for our task.

In this chapter, we will cover two topics that have recently grown...

Transfer learning

We humans have an amazing ability to learn, and then we take what we have learned and apply the knowledge to different types of tasks. The more closely related the new task is to tasks we already know, the easier it is for us to solve the new task. Basically, we never really have to start from scratch when learning something new.

However, neural networks aren't afforded this same luxury; they need to be trained from scratch for each individual task we want to apply them to. As we have seen in previous chapters, neural networks are very good at learning how to do one thing very well, and because they only learn what lies within an interpolation of the distribution they have been trained to recognize, they are unable to generalize their knowledge to deal with tasks beyond what they have encountered in the training dataset.

In addition, deep neural networks...

Meta learning

Meta learning—also known as learning to learn—is another fascinating topic within deep learning and is considered by many to be a promising path toward Artificial General Intelligence (AGI). For those of you who do not know what AGI is, it is when artificial intelligence reaches the capacity to understand and learn to do any type of intelligent task that a human is capable of doing, which is the goal of artificial intelligence.

Deep neural networks, as we know, are very data-hungry and require a lot of training time (depending on the size of the model), which can sometimes be several weeks, whereas humans are able to learn new concepts and skills a lot faster and more efficiently. For example, as kids, we can quickly learn to tell the difference between a donkey, a horse, and a zebra with absolute certainty after only seeing them once or a handful of...

Summary

In this chapter, we covered two very fascinating areas within the field of deep learning—transfer learning and meta learning—both of which hold the promise of furthering the field of not only deep learning but also artificial intelligence by enabling neural networks to learn additional tasks and generalize over unseen distributions. We explored several meta learning approaches, including model-based, metric-based, and optimization-based, and explored how they differ.

In the next chapter, we will learn about geometric deep learning.