An introduction to AutoML

The goal of AutoML is to enable domain experts who are unfamiliar with machine learning technologies to use ML techniques easily.

In this chapter, we will go through a practical exercise using Google Cloud, and do quite a bit of hands-on work after briefly discussing the fundamentals. We will talk about automatic data preparation, automatic feature engineering, and automatic model generation. Then, we introduce AutoKeras and Cloud AutoML with its multiple solutions for Table, Vision, Text, Translation, and for Video processing.

What is AutoML?

During the previous chapters we have introduced several models used in modern machine learning and deep learning. For instance, we have seen architectures such as Dense networks, CNNs, RNNs, Autoencoders, and GANs.

Two observations are in order. First, these architectures are manually designed by deep learning experts, and are not necessarily easy to explain to non-experts. Second, the composition of these architectures themselves was a manual process, which involved a lot of human intuition and trial and error.

Today, one primary goal of artificial intelligence research is to achieve Artificial General Intelligence (AGI) – the intelligence of a machine that can understand and automatically learn any type of work or activity that a human being can do. However, the reality was very different before AutoML research and industrial applications started. Indeed, before AutoML, designing deep learning architectures was very similar to crafting – the activity...

Achieving AutoML

How can AutoML achieve the goal of end-to-end automatization? Well, you are probably already guessing that a natural choice is to use machine learning – that's very cool – AutoML uses ML for automating ML pipelines.

What are the benefits? Automating the creation and tuning of the machine learning end-to-end offers produces simpler solutions, reduces the time to produce them, and ultimately might produce architectures that could potentially outperform the models that were crafted by hand.

Is this a closed research area? Quite the opposite. At the beginning of 2020, AutoML is a very open research field, which is not surprising, as the initial paper drawing attention to AutoML was published at the end of 2016.

Automatic data preparation

The first stage of a typical machine learning pipeline deals with data preparation (recall the pipeline of Figure 1). There are two main aspects that should be taken into account: data cleansing, and data synthesis.

Data cleansing is about improving the quality of data by checking for wrong data types, missing values, errors, and by applying data normalization, bucketization, scaling, and encoding. A robust AutoML pipeline should automate all of these mundane but extremely important steps as much as possible.

Data synthesis is about generating synthetic data via augmentation for training, evaluation, and validation. Normally, this step is domain-specific. For instance, we have seen how to generate synthetic CIFAR10-like images (Chapter 4, Convolutional Neural Networks) by using cropping, rotation, resizing, and flipping operations. One can also think about generate additional images or video via GANs (see Chapter 6, Generative Adversarial Networks)...

Automatic feature engineering

Featuring engineering is the second step of a typical machine learning pipeline (see Figure 1). It consists of three major steps: feature selection, feature construction, and feature mapping. Let's look at each of them in turn:

Feature selection aims at selecting a subset of meaningful features by discarding those that are providing little contribution to the learning task. In this context, meaningful is truly dependent on the application and the domain of your specific problem.

Feature construction has the goal of building new derived features, starting from the basic ones. Frequently, this technique is used to allow better generalization and to have a richer representation of the data.

Feature extraction aims at altering the original feature space by means of a mapping function. This can be implemented in multiple ways; for instance, it can use autoencoders (see Chapter 9, Autoencoders), PCA, or clustering (see Chapter 10, Unsupervised...

Automatic model generation

Model generation and hyperparameter tuning is the typical third macro-step of a machine learning pipeline (see Figure 1).

Model generation consists of creating a suitable model for solving specific tasks. For instance, you will probably use CNNs for visual recognition, and you will use RNNs for either time series analysis or for sequences. Of course, many variants are possible, each of which is manually crafted through a process of trial and error, and works for very specific domains.

Hyperparameter tuning happens once the model is manually crafted. This process is generally very computationally expensive and can significantly change the quality of the results in a positive way. That's because tuning the hyperparameters can help to optimize our model further.

Automatic model generation is the ultimate goal of any AutoML pipeline. How can this be achieved? One approach consists in generating the model by combining a set of primitive operations...

AutoKeras

AutoKeras [6] provides functions to automatically search for the architecture and hyperparameters of deep learning models. The framework uses Bayesian optimization for efficient neural architecture search. You can install the alpha version by using pip:

pip3 install autokeras # for 0.4 version
pip3 install git+git://github.com/keras-team/autokeras@master#egg=autokeras # for 1.0 version

The architecture is explained in Figure 3 (taken from [6]):

1. The user calls the API
2. The searcher generates neural architectures on CPU
3. Real neural networks with parameters on RAM from the neural architectures
4. The neural network is copied to GPU for training
5. Trained neural networks are saved on storage devices
6. The searcher is updated based on the training results

Steps 2 to 6 will repeat until a time limit is reached:

Figure 3: AutoKeras system overview

Google Cloud AutoML

Cloud AutoML (https://cloud.google.com/automl/) is a full suite of products for image, video, and text processing. As of the end of 2019, the suite consists of the following components, which do not require you to know how the deep learning networks are shaped internally:

AutoML Tables

• Enables you to automatically build and deploy state-of-the-art machine learning models on structured data used for general supervised classification and regression (see chapters 1, 2, and 3).

AutoML Vision

• AutoML Vision: Enables you to train machine learning models to classify your images according to your own defined labels.
• AutoML Object Detection: Used to automatically build a custom model to detect objects in an image with bounding boxes and labels, then deploy it to the cloud or on the edge.

AutoML Natural Language

• AutoML Text Classification: Used to automatically build a machine learning model to classify content into a custom set...

Bringing Google AutoML to Kaggle

On November 4th 2019, Google decided to integrate AutoML directly in Kaggle. To get started, you need to link your GCP account from Kaggle and authorize the access. This is easily done from a Kaggle Notebook as explained in Figure 76:

Figure 76: AutoML and Kaggle

The final step consists simply of activating AutoML (see Figure 77):

Figure 77: Activating AutoML from Kaggle

Summary

The goal of AutoML is to enable domain experts who are not familiar with machine learning technologies to use ML techniques easily. The primary goal is to reduce the steep learning curve and the huge costs of handcrafting machine learning solutions by making the whole end-to-end machine learning pipeline (data preparation, feature engineering, and automatic model generation) more automated.

After reviewing the state-of-the-art solution available at the end of 2019, we discussed how to use Cloud AutoML for text, videos, and images, achieving results comparable to the ones achieved with handcrafted models. AutoML is probably the fastest growing research topic and the interested reader can understand the latest results at https://www.automl.org/.

The next chapter discusses the math behind deep learning, a rather advanced topic that is recommended if you are interested in understanding what is going on "under the hood" when you play with neural networks.

References

1. Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le; 2016, http://arxiv.org/abs/1611.01578.
2. Efficient Neural Architecture Search via Parameter Sharing, Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, Jeff Dean, 2018, https://arxiv.org/abs/1802.03268.
3. Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents, Zalán Borsos, Andrey Khorlin, Andrea Gesmundo, 2019, https://arxiv.org/abs/1906.08102.
4. NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm, Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, Wolfgang Banzhaf, 2018 https://arxiv.org/abs/1810.03522.
5. Random Search for Hyper-Parameter Optimization, James Bergstra, Yoshua Bengio, 2012, http://www.jmlr.org/papers/v13/bergstra12a.html.
6. Auto-Keras: An Efficient Neural Architecture Search System, Haifeng Jin, Qingquan Song and Xia Hu, 2019, https://www.kdd.org/kdd2019/accepted-papers/view...