Understanding neural network gradients

The goal of machine learning for an MLP is to find the weights and biases that will effectively map the inputs to the desired outputs. The weights and biases generally get initialized randomly. In the training process, with a provided dataset, they get updated iteratively and objectively in batches to minimize the loss function, which uses gradients computed with a method called backward propagation, also known as backpropagation. A batch is a subset of the dataset used for training or evaluation, allowing the neural network to process the data in smaller groups rather than the entire dataset at once. The loss function is also known as the error function or the cost function.

Backpropagation is a technique to find out how sensitive a change of weights and bias of every neuron is to the overall loss by using the partial derivative of the loss with respect to the weights and biases. Partial derivatives from calculus are a measure of the rate...