Running HPO with Ray Tune using Optuna and HyperBand

Now, let's do some experiments with different search algorithms and schedulers. Given that Optuna is such a great TPE-based search algorithm, and ASHA is a great scheduler that does asynchronous parallel trials with early termination of the unpromising ones, it would be interesting to see how many changes we need to do to make this work.

It turns out the change is very minimal based on what we have already done in the previous section. Here, we will illustrate the four main changes:

1. Install the Optuna package. This can be done by running the following command:

   pip install optuna==2.10.0

This will install Optuna in the same virtual environment that we had before. If you have already run pip install -r requirements.text, then Optuna has already been installed and you can skip this step.

2. Import the relevant Ray Tune modules that integrate with Optuna and the ASHA scheduler (here, we use the HyperBand implementation...