Deep Neural Architecture Search

The previous chapters introduced and recapped different neural networks (NNs) that are designed to handle different types of data. Designing these networks requires knowledge and intuition that can only be gained by consuming years of research in the field. The bulk of these networks are hand-designed by experts and researchers. This includes inventing completely novel NN layers and constructing an actually usable architecture by combining and stacking NN layers that already exist. Both tasks require a ton of iterative experimentation time to burn to actually achieve success in creating a network that is useful.

Now, imagine a world where we can focus on inventing useful novel layers while the software takes care of automating the final architecture-building process. Automated architecture search methods help to accomplish exactly that by streamlining the task of designing the best final NN architecture, as long as appropriate search spaces are selected...

Technical requirements

This chapter includes practical implementation in the Python programming language. These simple methods will need to have the following libraries installed:

• numpy
• pytorch
• catalyst == 21.12
• scikit-learn

You can find the code files for this chapter on GitHub at https://github.com/PacktPublishing/The-Deep-Learning-Architect-Handbook/tree/main/CHAPTER_7.

Understanding the big picture of NAS

Before we dive into the details of the big picture of NAS methods, it’s important to note that although NAS minimizes the manual effort necessary for shaping the final architecture, it doesn’t completely negate the need for expertise in the field. As we discussed earlier, foundational knowledge in deep learning (DL) is crucial for selecting appropriate search spaces and interpreting the results of NAS accurately. Search spaces are the set of possible options or configurations that can be explored during a search. Furthermore, the performance of NAS heavily relies on the quality of the training data and the relevance of the search space to the task at hand. Therefore, domain expertise is still necessary to ensure that the final architecture is not only efficient but also accurate and relevant to the problem being solved. By the end of this section, you will have a better understanding of how to leverage your domain expertise to optimize...

Understanding general hyperparameter search-based NAS

In ML, parameters typically refer to the weights and biases that a model learns during training, while hyperparameters are values that are set before training begins and influence how the model learns. Examples of hyperparameters include learning rate and batch size. General hyperparameter search optimization algorithms are a type of NAS method to automatically search for the best hyperparameters to use for constructing a given NN architecture. Let’s go through a few of the possible hyperparameters. In a multi-layer perceptron (MLP), hyperparameters could be the number of layers that control the depth of the MLP, the width of each of the layers, and the type of intermediate layer activation used. In a CNN, hyperparameters could be the filter size of the convolutional layer, the stride size of each of the layers, and the type of intermediate layer activation used after each convolutional layer.

For NN architectures, the...

Understanding RL-based NAS

RL is a family of learning algorithms that deal with the learning of a policy that allows an agent to make consecutive decisions on its actions while interacting with states in an environment. Figure 7.3 shows a general overview of RL algorithms:

Figure 7.3 – General overview of RL algorithms

This line of algorithms is most popularly utilized to create intelligent bots for games that can act as offline players against real humans. In the context of a digital game, the environment represents the entire setting in which the agent operates, including aspects such as the position and status of the in-game character, as well as conditions of the in-game world. The state, on the other hand, is a snapshot of the environment at a given time, reflecting the current conditions of the game. One key component in RL is the environment feedback component that can provide either a reward or punishment. In digital games, examples of rewards...

Understanding non-RL-based NAS

The core of NAS is about intelligently searching through different child architecture configurations by making decisions based on prior search experience to find the best child architecture in a non-random and non-brute-force way. The core of RL, on the other hand, involves utilizing a controller-based system to achieve that intelligence. Intelligent NAS can be achieved without using RL, and in this section, we will go through a simplified version of the progressive growing-from-scratch style of NAS without a controller and another competitive version of elimination from a complex fully defined NN macroarchitecture and microarchitecture.

Understanding path elimination-based NAS

First and foremost, differentiable architecture search (DARTS) is a method that extends the DAG search space defined in ENAS by removing the RL controller component. Instead of choosing previous nodes to connect to and choosing which operation to use for a node, all operations...

Summary

NAS is a method that is generalized to any NN type, allowing for the automation of creating new and advanced NNs without the need for manual neural architecture design. As you may have guessed, NAS dominates the image-based field of NNs. The EfficientNet model family exemplifies the impact NAS provides to the image-based NN field. This is due to the inherent availability of a wide variety of CNN components that make it more complicated to design when compared to a simple MLP. For sequential or time-series data handling, there are not many variations of RNN cells, and thus the bulk of work in NAS for RNNs is focused on designing a custom recurrent cell. More work could have been done to accommodate transformers as it is the current state of the art, capable of being adapted to a variety of data modalities.

NAS is mainly adopted by researchers or practitioners in larger institutions. One of the key traits practitioners want when trying to train better models for their use...