Hyperparameter tuning to find the optimal parameters

In machine learning, we typically deal with parametric or non-parametric models. These models represent the distribution of the training data in order to make predictions for unseen data from the same distribution. While parametric models (such as linear regression, logistic regression, and neural networks) represent the training data distribution by using a learned set of parameters, non-parametric models describe the training data through other traits such as decision trees (all tree-based classifiers), training samples (k- nearest neighbors), or weighted training samples (support vector machine).

The Figure 9.1 outlines a few of the key differences between parametric and non- parametric models:

Figure 9.1: The difference between parametric and non-parametric models

The term hyperparameter refers to all parameters that are used to configure and tune the training process of parametric or non...