Deep Learning Fundamentals

In the previous chapter, we created some machine learning models using neural network packages in R. This chapter will look at some of the fundamentals of neural networks and deep learning by creating a neural network using basic mathematical and matrix operations. This application sample will be useful for explaining some key parameters in deep learning algorithms and some of the optimizations that allow them to train on large datasets. We will also demonstrate how to evaluate different hyper-parameters for models to find the best set. In the previous chapter, we briefly looked at the problem of overfitting; this chapter goes into that topic in more depth and looks at how you can overcome this problem. It includes an example use case using dropout, the most common regularization technique in deep learning.

This chapter covers the following topics:

...

Building neural networks from scratch in R

Although we have already used some neural network algorithms, it's time to dig a bit deeper into how they work. This section demonstrates how to code a neural network from scratch. It might surprise you to see that the core code for a neural network can be written in fewer than 80 lines! The code for this chapter does just that using an interactive web application written in R. It should give you more of an intuitive understanding of neural networks. First we will look at the web application, then we will delve more deeply into the code for the neural network.

Neural network web application

First, we will look at an R Shiny web application. I encourage you to run the application...

Using regularization to overcome overfitting

In the previous chapter, we saw the diminishing returns from further training iterations on neural networks in terms of their predictive ability on holdout or test data (that is, data not used to train the model). This is because complex models may memorize some of the noise in the data rather than learning the general patterns. These models then perform much worse when predicting new data. There are some methods we can apply to make our model generalize, that is, fit the overall patterns. These are called regularization and aim to reduce testing errors so that the model performs well on new data.

The most common regularization technique used in deep learning is dropout. However, we will also discuss two other regularization techniques that have a basis in regression and deep learning. These two regularization techniques are L1 penalty...

Use case – improving out-of-sample model performance using dropout

Dropout is a novel approach to regularization that is particularly valuable for large and complex deep neural networks. For a much more detailed exploration of dropout in deep neural networks, see Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinav, R. (2014). The concept behind dropout is actually quite straightforward. During the training of the model, units (for example, input and hidden neurons) are probabilistically dropped along with all connections to and from them.

For example, the following diagram is an example of what might happen at each step of training for a model where hidden neurons and their connections are dropped with a probability of 1/3 for each epoch. Once a node is dropped, its connections to the next layer are also dropped. In the the following diagram, the...

Summary

This chapter began by showing you how to program a neural network from scratch. We demonstrated the neural network in a web application created by just using R code. We delved into how the neural network actually worked, showing how to code forward-propagation, cost functions, and backpropagation. Then we looked at how the parameters for our neural network apply to modern deep learning libraries by looking at the mx.model.FeedForward.create function from the mxnet deep learning library.

Then we covered overfitting, demonstrating several approaches to preventing overfitting, including common penalties, the Ll penalty and L2 penalty, ensembles of simpler models, and dropout, where variables and/or cases are dropped to make the model noisy. We examined the role of penalties in regression problems and neural networks. In the next chapter, we will move into deep learning and...