Implementing Hyper Band

The extension of Successive Halving, the Hyper Band (HB) method (see Chapter 6), is implemented in the scikit-hyperband package. This package is built on top of sklearn, which means it also provides a very similar interface for GridSearchCV, RandomizedSearchCV, HalvingGridSearchCV, and HalvingRandomSearchCV.

In contrast with the default SH budget definition in the sklearn implementation, Scikit-Hyperband defines the budget as the number of estimators, n_estimators, in an ensemble of trees, or the number of iterations for estimators trained with stochastic gradient descent, such as the XGBoost algorithm. Additionally, we can use any other hyperparameters that exist in the estimator as the budget definition. However, scikit-hyperband doesn’t allow us to use the number of samples as the budget definition.

Let’s use the same example as in the Implementing Successive Halving section, but with a different hyperparameter space. Here, we use the...