An Introduction to AutoML

The goal of AutoML is to enable domain experts who are unfamiliar with machine learning technologies to use ML techniques easily.

In this chapter, we will go through a practical exercise using Google Cloud Platform and do quite a bit of hands-on work after briefly discussing the fundamentals.

We will cover:

• Automatic data preparation
• Automatic feature engineering
• Automatic model generation
• AutoKeras
• Google Cloud AutoML with its multiple solutions for table, vision, text, translation, and video processing

Let’s begin with an introduction to AutoML.

What is AutoML?

During the previous chapters, we introduced several models used in modern machine learning and deep learning. For instance, we have seen architectures such as dense networks, CNNs, RNNs, autoencoders, and GANs.

Two observations are in order. First, these architectures are manually designed by deep learning experts and are not necessarily easy to explain to non-experts. Second, the composition of these architectures themselves was a manual process, which involved a lot of human intuition and trial and error.

Today, one primary goal of artificial intelligence research is to achieve Artificial General Intelligence (AGI) – the intelligence of a machine that can understand and automatically learn any type of work or activity that a human being can do. It should be noted that many researchers do not believe that AGI is achievable because there is not only one form of intelligence but many forms.

Personally, I tend to agree with this view. See https:/...

Achieving AutoML

How can AutoML achieve the goal of end-to-end automatization? Well, you have probably already guessed that a natural choice is to use machine learning – that’s very cool. AutoML uses ML for automating ML pipelines.

What are the benefits? Automating the creation and tuning of machine learning end to end offers simpler solutions, reduces the time to produce them, and ultimately might produce architectures that could potentially outperform models that were crafted by hand.

Is this a closed research area? Quite the opposite. At the beginning of 2022, AutoML is a very open research field, which is not surprising, as the initial paper drawing attention to AutoML was published at the end of 2016.

Automatic data preparation

The first stage of a typical machine learning pipeline deals with data preparation (recall the pipeline in Figure 13.1). There are two main aspects that should be taken into account: data cleansing and data synthesis:

Data cleansing is about improving the quality of data by checking for wrong data types, missing values, and errors, and by applying data normalization, bucketization, scaling, and encoding. A robust AutoML pipeline should automate all of these mundane but extremely important steps as much as possible.

Data synthesis is about generating synthetic data via augmentation for training, evaluation, and validation. Normally, this step is domain-specific. For instance, we have seen how to generate synthetic CIFAR10-like images (Chapter 4) by using cropping, rotation, resizing, and flipping operations. One can also think about generating additional images or video via GANs (see Chapter 9) and using the augmented synthetic dataset for training...

Automatic feature engineering

Feature engineering is the second step of a typical machine learning pipeline (see Figure 13.1). It consists of three major steps: feature selection, feature construction, and feature mapping. Let’s look at each of them in turn:

Feature selection aims at selecting a subset of meaningful features by discarding those that are making little contribution to the learning task. In this context, “meaningful” truly depends on the application and the domain of your specific problem.

Feature construction has the goal of building new derived features, starting from the basic ones. Frequently, this technique is used to allow better generalization and to have a richer representation of the data.

Feature mapping aims at altering the original feature space by means of a mapping function. This can be implemented in multiple ways; for instance, it can use autoencoders (see Chapter 8), PCA (see Chapter 7), or clustering (see Chapter 7)...

Automatic model generation

Model generation and hyperparameter tuning is the typical third macro-step of a machine learning pipeline (see Figure 13.1).

Model generation consists of creating a suitable model for solving specific tasks. For instance, you will probably use CNNs for visual recognition, and you will use RNNs for either time series analysis or for sequences. Of course, many variants are possible, each of which is manually crafted through a process of trial and error and works for very specific domains.

Hyperparameter tuning happens once the model is manually crafted. This process is generally very computationally expensive and can significantly change the quality of the results in a positive way. That’s because tuning the hyperparameters can help to optimize our model further.

Automatic model generation is the ultimate goal of any AutoML pipeline. How can this be achieved? One approach consists of generating the model by combining a set of primitive operations...

AutoKeras

AutoKeras [6] provides functions to automatically search for architecture and hyperparameters of deep learning models. The framework uses Bayesian optimization for efficient neural architecture search. You can install the alpha version by using pip:

pip3 install autokeras # for 1.19 version

The architecture is explained in Figure 13.3 [6]:

Figure 13.3: AutoKeras system overview

The architecture follows these steps:

1. The user calls the API.
2. The searcher generates neural architectures on the CPU.
3. Real neural networks with parameters are built on RAM from the neural architectures.
4. The neural network is copied to the GPU for training.
5. The trained neural networks are saved on storage devices.
6. The searcher is updated based on the training results.

Steps 2 to 6 will repeat until a time limit is reached.

Google Cloud AutoML and Vertex AI

Google Cloud AutoML (https://cloud.google.com/automl/) is a full suite of products for image, video, and text processing. AutoML can be used to train high-quality custom machine learning models with minimal effort and machine learning expertise.

Vertex AI brings together the Google Cloud services for building ML under one, unified UI and API. In Vertex AI, you can now easily train, compare, test, and deploy models. Then you can serve a model with sophisticated ways to monitor and run experiments (see https://cloud.google.com/vertex-ai).

As of 2022, the suite consists of the following components, which do not require you to know how the deep learning networks are shaped internally:

Vertex AI

• Unified platform to help you build, deploy, and scale more AI models

Structured data

• AutoML Tables: Automatically build and deploy state-of-the-art machine learning models on structured data

Sight

• AutoML...

Summary

The goal of AutoML is to enable domain experts who are not familiar with machine learning technologies to use ML techniques easily. The primary goal is to reduce the steep learning curve and the huge costs of handcrafting machine learning solutions by making the whole end-to-end machine learning pipeline (data preparation, feature engineering, and automatic model generation) more automated.

After reviewing the state-of-the-art solution available at the end of 2022, we discussed how to use Google Cloud AutoML both for text, videos, and images, achieving results comparable to the ones achieved with handcrafted models. AutoML is probably the fastest-growing research topic and interested readers can find the latest results at https://www.automl.org/.

The next chapter discusses the math behind deep learning, a rather advanced topic that is recommended if you are interested in understanding what is going on “under the hood” when you play with neural networks...

References

1. Zoph, B., Le, Q. V. (2016). Neural Architecture Search with Reinforcement Learning. http://arxiv.org/abs/1611.01578
2. Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., Dean, J. (2018). Efficient Neural Architecture Search via Parameter Sharing. https://arxiv.org/abs/1802.03268
3. Borsos, Z., Khorlin, A., Gesmundo, A. (2019). Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents. https://arxiv.org/abs/1906.08102
4. Lu, Z., Whalen, I., Boddeti V., Dhebar, Y., Deb, K., Goodman, E., and Banzhaf, W. (2018). NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm. https://arxiv.org/abs/1810.03522
5. Bergstra, J., Bengio, Y. (2012). Random search for hyper-parameter optimization. http://www.jmlr.org/papers/v13/bergstra12a.html
6. Jin, H., Song, Q., and Hu, X. (2019). Auto-Keras: An Efficient Neural Architecture Search System. https://arxiv.org/abs/1806.10282
7. Faes, L., et al. (2019). Automated deep learning...