Backpropogation Is Just Steepest Descent with Automatic Differentiation (2013)

But we have two problems:

One thing we could try is bumping each parameter by a small amount to get partial derivatives numerically

But this would mean evaluating our function many times and moreover we could easily get numerical errors as a result of the vagaries of floating point arithmetic. For a complex function such as one which calculates the cost function of an ANN, there is a risk that we may end up getting a poor approximation for the derivative and thus a poor estimate for the parameters of the model. The pictorial representation of an ANN is effectively the data flow graph of the cost function (without the final cost calculation itself) and its differential can be calculated as just being identical to backpropagation.

Source: idontgetoutmuch.wordpress.com