A Practical Guide to Hyperparameter Optimization

Arguably the most important hyperparameter, the learning rate, roughly speaking, controls how fast your neural net “learns”. The Algorithm: Bayesian methods attempt to build a function (more accurately, a probability distribution over possible function) that estimates how good your model might be for a certain choice of hyperparameters. In essence, the left-hand side says that the probability that the true function that maps hyperparameters to the model’s metrics (like validation accuracy, log likelihood, test error rate, etc.) is \( F_n(X) \), given some sample data \(X_n\) is equal to whatever’s on the right-hand side.

Source: blog.nanonets.com