Model Optimizations – Hyperparameter Tuning and NAS

We have now become quite familiar with some of the Vertex AI offerings related to managing data, training no-code and low-code models, and launching large-scale custom model training jobs (with metadata tracking and monitoring capabilities). As ML practitioners, we know that it is highly unlikely that the first model we train would be the best model for a given use case and dataset. Thus, in order to find the best model (which is the most accurate and least biased), we often use different model optimization techniques. Hyperparameter Tuning (HPT) and Neural Architecture Search (NAS) are two such model optimization techniques. In this chapter, we will learn how to configure and launch model optimization experiments using Vertex AI on Google Cloud.

In this chapter, we will first learn about the importance of model optimization techniques such as HPT and then learn how to quickly set up and launch HPT jobs within Google Vertex...

Technical requirements

The code examples shown in this chapter can be found in the following GitHub repo: https://github.com/PacktPublishing/The-Definitive-Guide-to-Google-Vertex-AI/tree/main/Chapter09

What is HPT and why is it important?

Hyperparameter tuning, or HPT for short, is a popular model optimization technique that is very commonly used across ML projects. In this section, we will learn about hyperparameters, the importance of tuning them, and different methods of finding the best hyperparameters for a machine learning algorithm.

What are hyperparameters?

When we train an ML system, we basically have three kinds of data – input data, model parameters, and model hyperparameters. Input data refers to our training or test data that is associated with the problem we are solving. Model parameters are the variables that we modify during the model training process and we try to adjust them to fit the training data. Model hyperparameters, on the other hand, are variables that govern the training process itself. These hyperparameters are fixed before we start to train our model. For example, learning rate, optimizer, batch size, number of hidden layers in a neural network...

Setting up HPT jobs on Vertex AI

In this section, we will learn how to set up HPT jobs with Vertex AI. We will use the same neural network model experiment from Chapter 7, Training Fully Custom ML Models with Vertex AI, and optimize its hyperparameters to get the best model settings.

The first step is to create a new Jupyter Notebook in Vertex AI Workbench and import useful libraries:

import numpy as np
import glob
import matplotlib.pyplot as plt
import os
import google.cloud.aiplatform as aiplatform
from google.cloud.aiplatform import hyperparameter_tuning as hpt
from datetime import datetime
TIMESTAMP = datetime.now().strftime("%Y%m%d%H%M%S")
%matplotlib inline

Next, we set up project configurations:

PROJECT_ID='************'
REGION='us-west2'
SERVICE_ACCOUNT='[e-mail]'
BUCKET_URI='gs://my-training-artifacts'

Then, we initialize the Vertex AI SDK:

aiplatform.init(project=PROJECT_ID...

What is NAS and how is it different from HPT?

Artificial Neural Networks or ANNs are widely used today for solving complex ML problems. Most of the time, these network architectures are hand-designed by ML experts, which may not be optimal every time. Neural Architecture Search or NAS is a technique that automates the process of designing neural network architectures that usually outperform hand-designed networks.

Although both HPT and NAS are used as model optimization techniques, there are certain differences in how they both work. HPT assumes a given architecture and focuses on optimizing the hyperparameters that lead to the best model. HPT optimizes hyperparameters such as learning rate, optimizer, batch size, activation function, and so on. NAS, on the other hand, focuses on optimizing architecture-specific parameters (in a way, it automates the process of designing a neural network architecture). NAS optimizes parameters such as the number of layers, number of units, types...

NAS on Vertex AI overview

Vertex AI NAS is an optimization technique that can be leveraged to find the best neural network architecture for a given ML use case. NAS-based optimization searches for the best network in terms of accuracy but can also be augmented with other constraints such as latency, memory, or a custom metric as per the requirements. In general, the search space of possible neural networks can be quite large and NAS may support a search space as large as 10^20. In the past few years, NAS has been able to successfully generate some state-of-the-art computer vision network architectures, including NASNet, MNasNet, EfficientNet, SpineNet, NAS-FPN, and so on.

It may seem complex, but NAS features are quite flexible and easy to use. A beginner can leverage prebuilt modules for search spaces, trainer scripts, and Jupyter notebooks to start exploring Vertex AI NAS on a custom dataset. If you are an expert, you could potentially develop custom trainer scripts, custom search...

Summary

In this chapter, we discussed the importance of applying model optimization techniques to get the best performance for our application. We learned about two model optimization methods – HPT and NAS, with their similarities and differences. We also learned how to set up and launch large-scale HPT jobs on Vertex AI with code examples. Additionally, we discussed some best practices to get the best out of both HPT and NAS.

After reading this chapter, you should have a fair understanding of the term “model optimization” and its importance while developing ML applications. Additionally, you should now be confident about quickly setting up small to large-scale hyperparameter tuning experiments with the help of Vertex AI tooling on Google Cloud. You should also have a fair understanding of NAS, its differences from HPT, and the best practices for setting up a NAS job.

Now that we understand the importance and common methods of model optimization techniques...