Introduction to Learning Rates in Machine Learning

When training is initiated using a large learning rate, the loss doesn’t improve (and sometimes even increases) during the first few iterations. To investigate whether the learning rate is too large or too low, diagnostic plots such as the line plot of loss over training epochs can be used. The most basic example is to make the learning rate smaller once the performance of the model reaches a plateau, such as by decreasing the learning rate by a factor of two or an order of magnitude.

Source: heartbeat.fritz.ai