Chapter 7: Meta-SGD and Reptile Algorithms

1. Unlike MAML, in Meta-SGD, along with finding optimal parameter value, 
   
   , we also find the optimal learning rate, 
   
   , and update the direction.
2. The learning rate is implicitly implemented in the adaptation term. So, in Meta-SGD, we don't initialize a learning rate with a small scalar value. Instead, we initialize them with random values with the same shape as
   
    and learn them along with 
   
   .

3. The update equation of the learning rate can be expressed as 
   
   .
4. Sample n tasks and run SGD for fewer iterations on each of the sampled tasks, and then update our model parameter in a direction that is common to all the tasks.
5. The reptile update equation can be expressed as 
   
   .