”There is a similarity between knowing one’s way about a town and mastering a field of knowledge; from any given point one should be able to reach any other point. One is even better informed if one can immediately take the most convenient and quickest path from one point to the other.”

— George Pólya and Gábor Szegő, in the introduction of the legendary book Problems and Theorems in Analysis

The above quote is one of my all-time favorites. For me, it says that knowledge rests on many pillars. Like a chair has four legs, a well-rounded machine learning engineer also has a broad skill set that enables them to be effective in their job. Each of us focus on a balanced constellation of skills, and mathematics is a great addition for many. You can start machine learning without advanced mathematics, but at some point in your career, getting familiar with the mathematical background of machine learning can help you bring your skills to the next level.

There are two paths to mastery in deep learning. One starts from the practical parts and the other starts from theory. Both are perfectly viable, and eventually, they intertwine. This book is for those who started on the practical, application-oriented path, like data scientists, machine learning engineers, or even software developers interested in the topic.

This book is not a 100% pure mathematical treatise. At points, I will make some shortcuts to balance between clarity and mathematical correctness. My goal is to give you the “Eureka!” moments and help you understand the bigger picture instead of preparing you for a PhD in mathematics.

Most machine learning books I have read fall into one of two categories.

1. Focus on practical applications, but unclear and imprecise with mathematical concepts.
2. Focus on theory, involving heavy mathematics with almost no real applications.

I want this book to offer the best of both approaches: a sound introduction of basic and advanced mathematical concepts, keeping machine learning in sight at all times.

My goal is not only to cover the bare fundamentals but to give a breadth of knowledge. In my experience, to master a subject, one needs to go both deep and wide. Covering only the very essentials of mathematics would be like a tightrope walk. Instead of performing a balancing act every time you encounter a mathematical subject in the future, I want you to gain a stable footing. Such confidence can bring you very far and set you apart from others.

During our journey, we are going to follow a roadmap that takes us through

1. linear algebra,
2. calculus,
3. multivariable calculus,
4. and probability theory.

We are going to begin our journey with linear algebra. In machine learning, data is represented by vectors. Training a learning algorithm is the same as finding more descriptive representations of data through a series of transformations.

Linear algebra is the study of vector spaces and their transformations.

Simply put, a neural network is just a function that maps the data to a high-level representation. Linear transformations are the fundamental building blocks of these. Developing a good understanding of them will go a long way, as they are everywhere in machine learning.

While linear algebra shows how to describe predictive models, calculus has the tools to fit them to the data. When you train a neural network, you are almost certainly using gradient descent, a technique rooted in calculus and the study of differentiation.

Besides differentiation, its “inverse” is also a central part of calculus: integration. Integrals express essential quantities such as expected value, entropy, mean squared error, etc. They provide the foundations for probability and statistics.

However, when doing machine learning, we deal with functions with millions of variables. In higher dimensions, things work differently. This is where multivariable calculus comes in, where differentiation and integration are adapted to these spaces.

With linear algebra and calculus under our belt, we are ready to describe and train neural networks. However, we lack the understanding of extracting patterns from data. How do we draw conclusions from experiments and observations? How do we describe and discover patterns in them? These are answered by probability theory and statistics, the logic of scientific thinking. In the final chapter, we extend the classical binary logic and learn to deal with uncertainty in our predictions.

Mathematics follows a definition-theorem-proof structure that might be difficult to follow at first. If you are unfamiliar with such a flow, don’t worry. I’ll give a gentle introduction right now.

In essence, mathematics is the study of abstract objects (such as functions) through their fundamental properties. Instead of empirical observations, mathematics is based on logic, making it universal. If we want to use the powerful tool of logic, the mathematical objects need to be precisely defined. Definitions are presented in boxes like this below.

Given a definition, results are formulated as if A, then B statements, where A is the premise, and B is the conclusion. Such results are called theorems. For instance, if a function is differentiable, then it is also continuous. If a function is convex, then it has global minima. If we have a function, then we can approximate it with arbitrary precision using a single-layer neural network. You get the pattern. Theorems are the core of mathematics.

We must provide a sound logical argument to accept the validity of a proposition, one that deduces the conclusion from the premise. This is called a proof, responsible for the steep learning curve of mathematics. Contrary to other scientific disciplines, proofs in mathematics are indisputable statements, set in stone forever. On a practical note, look out for these boxes.

To enhance the learning experience, I’ll often make good-to-know but not absolutely essential information into remarks.

The most effective way of learning is building things and putting theory into practice. In mathematics, this is the only way to learn. What this means is that you need to read through the text carefully. Don’t take anything for granted just because it is written down. Think through every sentence. Take every argument and calculation apart. Try to prove theorems by yourself before reading the proofs.

With that in mind, let’s get to it! Buckle up for the ride; the road is long and full of twists and turns.

There are a number of text conventions used throughout this book. CodeInText indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, or URLs. For example: “Slicing works by specifying the first and last elements with an optional step size, using the syntax object[first:last:step].”

A block of code is set as follows:

from sklearn.datasets import load_iris 
data = load_iris() 
x, y = data["data"], data["target"] 
x[:10]

Any command-line input or output is written as follows:

(3.5, -2.71, 'a string')

Italics indicate new concepts or emphasis. For instance, words in menus or dialog boxes appear in the text like this. For example: "This is our first example of a non-differentiable function."

Chapter 1, Vectors and vector spaces covers what vectors are and how to work with them. We’ll travel from concrete examples through precise mathematical definitions to implementations, understanding vector spaces and NumPy arrays, which are used to represent vectors efficiently. Besides the fundamentals, we’ll learn

Chapter 2, The geometric structure of vector spaces moves forward by studying the concept of norms, distances, inner products, angles, and orthogonality, enhancing the algebraic definition of vector spaces with some much-needed geometric structure. These are not just tools for visualization; they play a crucial role in machine learning. We’ll also encounter our first algorithm, the Gram-Schmidt orthogonalization method, turning any set of vectors into an orthonormal basis.

In Chapter 3, Linear algebra in practice, we break out NumPy once more, and implement everything that we’ve learned so far. Here, we learn how to work with the high-performance NumPy arrays in practice: operations, broadcasting, functions, culminating in the from-scratch implementation of the Gram-Schmidt algorithm. This is also the first time we encounter matrices, the workhorses of linear algebra.

Chapter 4, Linear transformations is about the true nature of matrices; that is, structure-preserving transformations between vector spaces. This way, seemingly arcane things – such as the definition of matrix multiplication – suddenly make sense. Once more, we take the leap from algebraic structures to geometric ones, allowing us to study matrices as transformations that distort their underlying space. We’ll also look at one of the most important descriptors of matrices: the determinants, describing how the underlying linear transformations affect the volume of the spaces.

Chapter 5, Matrices and equations presents the third (and for us, the final) face of matrices as systems of linear equations. In this chapter, we first learn how to solve systems of linear equations by hand using the Gaussian elimination, then supercharge it via our newfound knowledge of linear algebra, obtaining the mighty LU decomposition. With the help of the LU decomposition, we go hard and achieve a roughly 70000 × speedup on computing determinants.

Chapter 6 introduces two of the most important descriptors of matrices: eigenvalues and eigenvectors. Why do we need them?

Because in Chapter 7, Matrix factorizations, we are able to reach the pinnacle of linear algebra with their help. First, we show that real and symmetric matrices can be written in diagonal form by constructing a basis from their eigenvectors, known as the spectral decomposition theorem. In turn, a clever application of the spectral decomposition leads to the singular value decomposition, the single most important result of linear algebra.

Chapter 8, Matrices and graphs closes the linear algebra part of the book by studying the fruitful connection between linear algebra and graph theory. By representing matrices as graphs, we are able to show deep results such as the Frobenius normal form, or even talk about the eigenvalues and eigenvectors of graphs.

In Chapter 9, Functions, we take a detailed look at functions, a concept that we have used intuitively so far. This time, we make the intuition mathematically precise, learning that functions are essentially arrows between dots.

Chapter 10, Numbers, sequences, and series continues down the rabbit hole, looking at the concept of numbers. Each step from natural numbers towards real numbers represents a conceptual jump, peaking at the study of sequences and series.

With Chapter 11, Topology, limits, and continuity, we are almost at the really interesting parts. However, in calculus, the objects, concepts, and tools are most often described in terms of limits and continuous functions. So, we take a detailed look at what they are.

Chapter 12 is about the single most important concept in calculus: Differentiation. In this chapter, we learn that the derivative of a function describes 1) the slope of the tangent line, and 2) the best local linear approximation to a function. From a practical side, we also look at how derivatives behave with respect to operations, most importantly the function composition, yielding the essential chain rule, the bread and butter of backpropagation.

After all the setup, Chapter 13, Optimization introduces the algorithm that is used to train virtually every neural network: gradient descent. For that, we learn how the derivative describes the monotonicity of functions and how local extrema can be characterized with the first and second order derivatives.

Chapter 14, Integration wraps our study of univariate functions. Intuitively speaking, integration describes the (signed) area under the functions’ graph, but upon closer inspection, it also turns out to be the inverse of differentiation. In machine learning (and throughout all of mathematics, really), integrals describe various probabilities, expected values, and other essential quantities.

Now that we understand how calculus is done in single variables, Chapter 15 leads us to the world of Multivariable functions, where machine learning is done. There, we have an entire zoo of functions: scalar-vector, vector-scalar, and vector-vector ones.

In Chapter 16, Derivatives and gradients, we continue our journey, overcoming the difficulties of generalizing differentiation to multivariable functions. Here, we have three kinds of derivatives: partial, total, and directional; resulting in the gradient vector and the Jacobian and Hessian matrices.

As expected, optimization is also slightly more complicated in multiple variables. This issue is cleared up by Chapter 17, Optimization in multiple variables, where we learn the analogue of the univariate second-derivative test, and implement the almighty gradient descent in its final form, concluding our study of calculus.

Now that we have a mechanistic understanding of machine learning, Chapter 18, What is probability? shows us how to reason and model under uncertainty. In mathematical terms, probability spaces are defined by the Kolmogorov axioms, and we’ll also learn the tools that allow us to work with probabilistic models.

Chapter 19 introduces Random variables and distributions, allowing us not only to bring the tools of calculus into probability theory, but to compact probabilistic models into sequences or functions.

Finally, in Chapter 20, we learn the concept of The expected value, quantifying probabilistic models and distributions with averages, variances, covariances, and entropy.

The code for this book is provided in the form of Jupyter notebooks, hosted on GitHub at https://github.com/cosmic-cortex/mathematics-of-machine-learning-book. To run the notebooks, you’ll need to install the required packages.

The easiest way to install them is using Conda. Conda is a great package manager for Python. If you don’t have Conda installed on your system, the installation instructions can be found here: https://bit.ly/InstallConda.

Note that Conda’s license might have some restrictions for commercial use. After installing Conda, follow the environment installation instructions in the book’s repository README.md.

Download the example code files

The code bundle for the book is hosted on GitHub at https://github.com/cosmic-cortex/mathematics-of-machine-learning-book. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781837027873.

Get in touch

Feedback from our readers is always welcome. General feedback: Email feedback@packtpub.com and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at questions@packtpub.com. Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packtpub.com/submit-errata, click Submit Errata, and fill in the form. Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packtpub.com with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Share your thoughts

Once you’ve read Mathematics of Machine Learning, we’d love to hear your thoughts! Click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere? Is your eBook purchase not compatible with the device of your choice?

Don’t worry; with every Packt book, you now get a DRM-free PDF version of that book at no cost.

Read anywhere, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there! You can get exclusive access to discounts, newsletters, and great free content in your inbox daily.

Follow these simple steps to get the benefits:

1. Scan the QR code or visit the link below:

   

   https://packt.link/free-ebook/9781837027873

2. Submit your proof of purchase.
3. That’s it! We’ll send your free PDF and other benefits to your email address directly.