Introduction to neural networks

Artificial neural networks (briefly, "nets" or ANNs) represent a class of machine learning models loosely inspired by studies about the central nervous systems of mammals. Each ANN is made up of several interconnected "neurons," organized in "layers." Neurons in one layer pass messages to neurons in the next layer (they "fire," in jargon terms) and this is how the network computes things. Initial studies were started in the early 50's with the introduction of the "perceptron" [1], a two-layer network used for simple operations, and further expanded in the late 60's with the introduction of the "back-propagation" algorithm used for efficient multi-layer network training (according to [2], [3]).

Some studies argue that these techniques have roots dating further back than normally cited[4].

Neural networks were a topic of intensive academic studies up until the 80's, at which point other, simpler approaches became more relevant. However, there has been a resurgence of interest starting in the mid 2000's, mainly thanks to three factors: a breakthrough fast learning algorithm proposed by G. Hinton [3], [5], [6]; the introduction of GPUs around 2011 for massive numeric computation; and the availability of big collections of data for training.

These improvements opened the route for modern "deep learning," a class of neural networks characterized by a significant number of layers of neurons that are able to learn rather sophisticated models based on progressive levels of abstraction. People began referring to it as "deep" when it started utilizing 3-5 layers a few years ago. Now, networks with more than 200 layers are commonplace!

This learning via progressive abstraction resembles vision models that have evolved over millions of years within the human brain. Indeed, the human visual system is organized into different layers. First, our eyes are connected to an area of the brain named the visual cortex (V1), which is located in the lower posterior part of our brain. This area is common to many mammals and has the role of discriminating basic properties like small changes in visual orientation, spatial frequencies, and colors.

It has been estimated that V1 consists of about 140 million neurons, with tens of billions of connections between them. V1 is then connected to other areas (V2, V3, V4, V5, and V6) doing progressively more complex image processing and recognizing more sophisticated concepts, such as shapes, faces, animals, and many more. It has been estimated that there are ~16 billion human cortical neurons and about 10-25% of the human cortex is devoted to vision [7]. Deep learning has taken some inspiration from this layer-based organization of the human visual system: early artificial neuron layers learn basic properties of images while deeper layers learn more sophisticated concepts.

This book covers several major aspects of neural networks by providing working nets in TensorFlow 2.0. So, let's start!