Optimizing the Model through Hyperparameter Tuning

Neural networks constitute multiple parameters that can affect the ultimate accuracy in predicting an event or a label. The typical parameters include:

• Batch size used for training
• Number of epochs
• Learning rate
• Number of hidden layers
• Number of hidden units in each hidden layer
• The activation function applied in the hidden layer
• The optimizer used

From the preceding list, we can see that the number of parameters that can be tweaked is very high. This makes finding the optimal combination of hyperparameters a challenge. Hyperparameter tuning as a service provided by Cloud ML Engine comes in handy in such a scenario.

In this chapter, we will go through:

• Why hyperparameter tuning is required
• An overview of how hyperparameter tuning works
• Implementing hyperparameter tuning in the cloud

...

The intuition of hyperparameter tuning

In order to gain a practical intuition of the need for hyperparameter tuning, let's go through the following scenario in predicting the accuracy of a given neural network architecture on the MNIST dataset:

• Scenario 1: High number of epochs and low learning rate
• Scenario 2: Low number of epochs and high learning rate

Let us create the train and test datasets in a Google Cloud environment, as follows:

1. Download the dataset:

mkdir data
curl -O https://s3.amazonaws.com/img-datasets/mnist.pkl.gz
gzip -d mnist.pkl.gz
mv mnist.pkl data/            

The preceding code creates a new folder named data, downloads the MNIST dataset, and moves it into the data folder.

2. Open Python in Terminal and import the required packages:

from __future__ import print_function 
import tensorflow as tf
import pickle # for handling the new data source
import numpy...

Summary

In this chapter, we understood how different parameter combinations affect the final accuracy measure and how hyperparameter tuning using Cloud ML engine helps in improving the accuracy further.

In the next chapter, we will learn how to identify overfitting and make our models more robust to previously-unseen data by setting the right parameters and defining the proper architectures.