Understanding the matrix norm
The norm of a matrix is a scalar value that measures the magnitude of the matrix. Therefore, the norm is a way to measure the size or length of a vector or a matrix. For example, the weights of a deep neural network are stored in matrices, and we would typically constrain the norm of the weights to be small to prevent overfitting. This allows us to quantify the magnitude, which is useful when comparing different vectors or matrices, which often consist of multiple elements. As it generalizes from the vector norm, we will first go through the basics of the vector norm.
Understanding the vector norm
Suppose we have a vector, a = [1,0, − 1], and another vector, b = [1,2,0]. To assess the similarity between these two vectors, we can argue that they are the same in the first element only and different for the remaining two elements. To compare these two vectors holistically, we need a single metric – one that summarizes the whole vector...
