4.1 What is a linear transformation?

With the introduction of inner products, orthogonality, and orthogonal/orthonormal bases, we know everything about the structure of our feature spaces. However, in machine learning, our interest mainly lies in transforming the data.

From this viewpoint, a neural network is just a function composed of smaller parts (known as layers), transforming the data to a new feature space in every step. One of the key components of models in machine learning are linear transformations.

You probably encountered them as functions of the form f(x) = Ax, but this is only one way to look at them. This section will start from a geometric viewpoint, then move towards the algebraic representation that you are probably already familiar with. To understand how neural networks can learn powerful high-level representations of the data, looking at the geometry of transforms is essential.

So, what linear transformations are? Let’s not hesitate a moment further, and...