Deep Dive into Math Behind Deep Networks

Deep Dive into Math Behind Deep Networks
Mysteries of Neural Networks Part I
Nowadays, having at our disposal many high-level, specialized libraries and frameworks such as Keras, TensorFlow or PyTorch, we do not need to constantly worry about the size of our weights matrices or remembers formula for the derivative of activation function we decided to use. Diagrams of the most popular activation functions together with their derivatives.Loss function
The basic source of information on the progress of the learning process is the value of the loss function. In each iteration we will calculate the values of the loss function partial derivatives with respect to each of the parameters of our neural network.

Source: towardsdatascience.com