Chapter 9: Working with Multimodal and Multitasking Data

In this chapter, we will learn how to use the AutoModel API to handle multimodal and multitasking data.

By the end of this chapter, you will have learned how to use the concepts and tools necessary to create models with multiple inputs and multiple outputs. You will be able to apply these concepts to your own projects by creating a model from scratch or by adapting the practical example shown in this chapter to other, similar datasets.

In this chapter, we will cover the following topics:

• Exploring models with multiple input or outputs
• Creating a multitasking/multimodal model
• Customizing the search space

But first, let's explain the technical requirements for this chapter.

Technical requirements

All the code examples in this book are available as Jupyter notebooks that can be downloaded from https://github.com/PacktPublishing/Automated-Machine-Learning-with-AutoKeras.

Since code cells can be executed, each notebook can be self-installed; simply add the code snippet that contains the requirements you need. For this reason, at the beginning of each notebook, there is a code cell for environment setup that installs AutoKeras and its dependencies.

So, to run the code examples in this chapter, you only need a computer with Ubuntu Linux as its OS and must install the Jupyter Notebook with the following line of code:

$ apt-get install python3-pip jupyter-notebook

Alternatively, you can also run these notebooks using Google Colaboratory, in which case you will only need a web browser. See Chapter 2, AutoKeras with Google Colaboratory, for more details. Furthermore, in the Installing AutoKeras section of that chapter, you will find other installation...

Exploring models with multiple inputs or outputs

As we will see later, sometimes, it may interest us that our model feeds on information from different sources (multimodal) and/or predicts multiple targets at the same time (multitask). AutoKeras has a class called AutoModel that allows us to define several sources and targets as a list of parameters. Let's dive a little deeper into this before looking at a practical example.

What is AutoModel?

AutoModel is a class that allows us to define a model in a granular way by defining not only its inputs and outputs but also its intermediate layers.

It can be used in two different ways:

• Basic: Here, the input/output nodes are specified and AutoModel infers the remaining part of the model.
• Advanced: Here, the high-level architecture is defined by connecting the layers (blocks) with the Functional API, which is the same as the Keras functional API.

Let's look at an example of each one.

Basic example

...

Creating a multitask/multimodal model

Based on the example provided at the beginning of this chapter, the model that we are going to create will take an image and its structured data attributes as input and will predict a category value and a scalar value. In this case, instead of using a dataset, we will generate our own data. The notebook we will be using that contains the complete source code can be found at https://github.com/PacktPublishing/Automated-Machine-Learning-with-AutoKeras/blob/main/Chapter09/Chapter9_MultiModel.ipynb.

Now, let's have a look at the relevant cells of the notebook in detail:

• Installing AutoKeras: As we've mentioned in the previous chapters, this snippet at the top of the notebook is responsible for installing AutoKeras and its dependencies using the pip package manager:

  !pip3 install autokeras

• Importing the necessary packages: The following lines load TensorFlow, the built-in Keras Reuters dataset, numpy, and AutoKeras as the necessary...

Customizing the search space

As we mentioned at the beginning of this chapter, there is an advanced way to use AutoModel. We can do this by defining the whole model architecture by connecting the layers (blocks) with the functional API, which is the same as the Keras functional API.

Let's do this in the following example:

input_node1 = ak.ImageInput()
output_node = ak.Normalization()(input_node1)
output_node = ak.ImageAugmentation()(output_node)
output_node1 = ak.ConvBlock()(output_node)
output_node2 = ak.ResNetBlock(version='v2')(output_node)
output_node1 = ak.Merge()([output_node1, output_node2])
 
input_node2 = ak.StructuredDataInput()
output_node = ak.CategoricalToNumerical()(input_node2)
output_node2 = ak.DenseBlock()(output_node)
 
output_node = ak.Merge()([output_node1, output_node2])
output_node1 = ak.ClassificationHead()(output_node)
output_node2 = ak.RegressionHead()(output_node)
 
model = ak.AutoModel(
    inputs=[input_node1, input_node2...

Summary

In this chapter, we learned what a multitasking model is, what a multimodal model is, and how to use the powerful AutoModel class to create efficient models with multiple inputs and outputs. You are now ready to apply these concepts to your own multimodel projects by creating them from scratch or by adapting this practical example for your own datasets.

In the next chapter, we will learn how to export our models and how to use a powerful visualization tool to track and visualize metrics such as loss and accuracy in real-time graphs.