10 New Things I Learnt from fast.ai v3

Training
Photo by Victor Freitas from PexelsThis section looks into a combination of tweaks for:
Weight initialisation
Hyperparameter setting
Model fitting/fine-tuning
Other improvements

Transfer learning
Model weights can either be (i) randomly initialised, or (ii) transferred from a pre-trained model in a process called transfer learning. So increasing the learning rate at the start helps the model to jump out away from that trench, explore the function surface and try to find areas where the loss is low and the region is not bumpy (because if it’s bumpy, it gets kicked out again). Fig. 6.2: Screenshot from course.fast.aiDiscriminative learning rates for pre-trained models
Train earlier layer(s) with super low learning rate, and train later layers with higher learning rate.

Source: towardsdatascience.com