Simple models with small datasets can often be fit in a single step, while larger datasets and more complex models must be fit by repeatedly using the model with training data and comparing the output with the expected label. If the prediction is accurate enough, we consider the model trained. If not, we adjust the model slightly and loop again.
Hyperparameters are values that change the way that the model is fit during these loops. Learning rate, for example, is a hyperparameter that sets how much a model is adjusted during each training cycle. A high learning rate means a model can be trained faster; but if it’s too high, the adjustments can be so large that the model is never “finely tuned” and not optimal.So how do you know what hyperparameter values you should use? Well, in the absence of a deep understanding of how the underlying algorithm works, you’ll need to experiment.
SciKit-Learn provides a way to tune hyperparameters by trying multiple combinations and finding the best result for a given performance metric. For example, a grid search approach tries combinations from a grid of possible values for the learning_rate and n_estimators hyperparameters of the GradientBoostingRegressor estimator.