Optimization

Optimization is a branch of applied mathematics that has applications in a multitude of fields, such as physics, engineering, economics, and so on, and is of vital importance in developing and training of deep neural networks. In this chapter, a lot of what we covered in previous chapters will be very relevant, particularly linear algebra and calculus.

As we know, deep neural networks are developed on computers and are, therefore, expressed mathematically. More often than not, training deep learning models comes down to finding the correct (or as close to the correct) set of parameters. We will learn more about this as we progress further through this book.

In this chapter, we'll mainly learn about two types of continuous optimization—constrained and unconstrained. However, we will also briefly touch on other forms of optimization, such as genetic algorithms...

Understanding optimization and it's different types

In optimization, our goal is to either minimize or maximize a function. For example, a business wants to minimize its costs while maximizing its profits or a shopper might want to get as much as possible while spending as little as possible. Therefore, the goal of optimization is to find the best case of , which is denoted by x^* (where x is a set of points), that satisfies certain criteria. These criteria are, for our purposes, mathematical functions known as objective functions.

For example, let's suppose we have the equation. If we plot it, we get the following graph:

You will recall from Chapter 1, Vector Calculus, that we can find the gradient of a function by taking its derivative, equating it to 0, and solving for x. We can find the point(s) at which the function has a minimum or maximum, as follows:

After...

Exploring the various optimization methods

Now that you know what optimization is, it's time to explore some of the methods used in practice. We will not be covering the entire field of optimization because that would require an entire book to cover. We will only cover the essential optimization methods that are applicable to deep learning.

Least squares

Least squares is a subclass of convex optimization. It is classified as having no constraints and takes the following form:

Here, , are rows of A, and is our optimization variable.

We can also express this as a set of linear equations of the form. Therefore, .

The problem of least squares is very similar to that of maximum likelihood estimation.

...

Exploring population methods

So far, we have dealt with optimization problems where we have a ball or particle that we edge along the curved space gradually and move toward the minima using gradient descent or Newton's method. Now, however, we will take a look at another class of optimization, where we use a population of individuals.

We spread these individuals across the optimization space, which prevents the optimization algorithm from getting stuck at local minima or a saddle point. These individuals can share information with each other about the local area they're in and use this to find an optimal solution that minimizes our function.

With these algorithms, we have an initial population and we would like to distribute them so that we cover as much ground as we can to give us the best chance of finding a globally optimal region.

We can sample our population from...

Summary

In this chapter, we covered a number of different kinds of optimization, such as convex and non-convex optimization, as well as what makes optimization such a challenging problem. We also had a look at how to define an optimization problem and explored a variety of methods, including population methods, simulated annealing, and gradient descent-based methods. In later chapters, we'll come to understand how optimization is used in deep learning and why it is such an important field for us to understand.

In the next chapter, we will learn about graph theory and its uses in the field to solve various problems.