Chapter 2. Neural Networks

In the previous chapter, we described several machine learning algorithms and we introduced different techniques to analyze data to make predictions. For example, we suggested how machines can use data of home selling prices to make predictions on the price for new houses. We described how large companies, such as Netflix, use machine learning techniques in order to suggest to users new movies they may like based on movies they have liked in the past, using a technique that is widely utilized in e-commerce by giants such as Amazon or Walmart. Most of these techniques, however, necessitate labeled data in order to make predictions on new data, and, in order to improve their performance, need humans to describe the data in terms of features that make sense.

Humans are able to quickly extrapolate patterns and infer rules without having the data cleaned and prepared for them. It would then be desirable if machines could learn to do the same. As we have discussed, Frank...

Why neural networks?

Neural networks have been around for many years, and they have gone through several periods during which they have fallen in and out of favor. However, in recent years, they have steadily gained ground over many other competing machine learning algorithms. The reason for this is that advanced neural net architecture has shown accuracy in many tasks that has far surpassed that of other algorithms. For example, in the field of image recognition, accuracy may be measured against a database of 16 million images named ImageNet.

Prior to the introduction of deep neural nets, accuracy had been improving at a slow rate, but after the introduction of deep neural networks, accuracy dropped from an error rate of 40% in 2010 to less than 7% in 2014, and this value is still falling. The human recognition rate is still lower, but only at about 5%. Given the success of deep neural networks, all entrants to the ImageNet competition in 2013 used some form of deep neural network. In addition...

Fundamentals

In the first chapter, we talked about three different approaches to machine learning: supervised learning, unsupervised learning, and reinforcement learning. Classical neural networks are a type of supervised machine learning, though we will see later that deep learning popularity is instead due to the fact that modern deep neural networks can be used in unsupervised learning tasks as well. In the next chapter, we will highlight the main differences between classical shallow neural networks and deep neural nets. For now, however, we will mainly concentrate on classical feed-forward networks that work in a supervised way. Our first question is, what exactly is a neural network? Probably the best way to interpret a neural network is to describe it as a mathematical model for information processing. While this may seem rather vague, it will become much clearer in the next chapters. A neural net is not a fixed program, but rather a model, a system that processes information, or...

Summary

In this chapter, we have introduced neural networks in detail and we have mentioned their success over other competing algorithms. Neural networks are comprised of the "units", or neurons, that belong to them or their connections, or weights, that characterize the strength of the communication between different neurons and their activity functions, that is, how the neurons process the information. We have discussed how we can create different architectures, and how a neural network can have many layers, and why inner (hidden) layers are important. We have explained how the information flows from the input to the output by passing from each layer to the next based on the weights and the activity function defined, and finally we have shown how we can define a method called back-propagation to "tune" the weights to improve the desired level of accuracy. We have also mentioned many of the areas where neural networks are and have been employed.

In the next chapter, we will continue discussing...