Using Deep Learning to Solve Binary Classification Problems

In this chapter, we will use Keras and TensorFlow to solve a tricky binary classification problem. We will start by talking about the benefits and drawbacks of deep learning for this type of problem, and then we will go right into developing a solution using the same framework we established in Chapter 2, Using Deep Learning to Solve Regression Problems. Finally, we will cover Keras callbacks in greater depth and even use a custom callback to implement a per epoch receiver operating characteristic / area under the curve (ROC AUC) metric.

We will cover the following topics in this chapter:

• Binary classification and deep neural networks
• Case study – epileptic seizure recognition
• Building a binary classifier in Keras
• Using the checkpoint callback in Keras
• Measuring ROC AUC in a custom callback
• Measuring precision...

Binary classification and deep neural networks

Binary classification problems, such as regression problems, are very common machine learning tasks. So much so that any book on deep learning wouldn't be complete without covering them. To be sure, we haven't really hit the sweet spot of deep neural networks quite yet, but we're well on our way. Before we get to the code, let's talk about the trade-offs you should consider when choosing a deep neural network to solve this kind of problem.

Benefits of deep neural networks

When compared to a more traditional classifier such as a logistic regression model, or even a tree-based model such as random forest or a gradient boosting machine, deep neural networks have...

Case study – epileptic seizure recognition

As you've probably guessed, we are going to be solving a binary classification problem. We will start by planning the problem using the same framework we established in Chapter 2, Using Deep Learning to Solve Regression Problems, and modify it as needed. You can find the complete code for this chapter in the book's GitHub repository, under Chapter 4, Using Deep Learning to Solve Regression Problems.

Defining our dataset

The dataset that we will be working on this chapter is called the Epileptic Seizure Recognition dataset. The data originally comes from a paper titled Indications of nonlinear deterministic and finite dimensional structures in time series of brain...

Building a binary classifier in Keras

Now that we've defined our problem, our inputs, our desired output, and our cost function, we can quickly code the rest in Keras. The only thing we're missing is a network architecture. We will talk more about that soon. One of my favorite things about Keras is how easy it is tune the network architecture. As you're about to see, it might take a lot of experimentation before you locate the best architecture. If that's true, a framework that easily changes makes your job easier!

The input layer

As before, our input layer needs to know the dimensions of our dataset. I like to build the entire Keras model inside a function, and allow that function to pass back the compiled...

Using the checkpoint callback in Keras

In Chapter 2, Using Deep Learning to Solve Regression Problems, we saw the .save() method, that allowed us to save our Keras model after we were done training. Wouldn't it be nice, though, if we could write our weights to disk every now and then so that we could go back in time in the preceding example and save a version of the model before it started to overfit? We could then stop right there and use the lowest variance version of the network.

That's exactly what the ModelCheckpoint callback does for us. Let's take a look:

checkpoint_callback = ModelCheckpoint(filepath="./model-weights.{epoch:02d}-{val_acc:.6f}.hdf5", monitor='val_acc', verbose=1, save_best_only=True)

What ModelCheckpoint will do for us is save our model at scheduled intervals. Here, we are telling ModelCheckpoint to save a copy of the...

Measuring ROC AUC in a custom callback

Let's use one more callback. This time, we will build a custom callback that computes Receiver Operating Characteristic Area Under the Curve (ROC AUC) at the end of every epoch, on both training and testing sets.

Creating a custom callback in Keras is actually really simple. All we need to do is create a class, inherent Callback, and override the method we need. Since we want to calculate the ROC AUC score at the end of each epoch, we will override on _epoch_end:

from keras.callbacks import Callback

class RocAUCScore(Callback):
    def __init__(self, training_data, validation_data):
        self.x = training_data[0]
        self.y = training_data[1]
        self.x_val = validation_data[0]
        self.y_val = validation_data[1]
        super(RocAUCScore, self).__init__()

    def on_epoch_end(self, epoch, logs={}):
        y_pred = self.model...

Measuring precision, recall, and f1-score

As you're likely experienced with other binary classifiers, I thought it was wise to take a few sentences to talk about how to create some of the normal metrics used with more traditional binary classifiers.

One difference between the Keras functional API and what you might be used to in scikit-learn is the behavior of the .predict() method. When using Keras, .predict() will return an nxk matrix of k class probabilities for each of the n classes. For a binary classifier, there will be only one column, the class probability for class 1. This makes the Keras .predict() more like the .predict_proba() in scikit-learn.

When calculating precision, recall, or other class-based metrics, you'll need to transform the .predict() output by choosing some operating point, as shown in the following code:

def class_from_prob(x, operating_point...

Summary

In this chapter, we talked about using deep neural networks as binary classifiers. We spent quite a bit of time talking about network architecture design choices and touched on the idea that searching and experimentation is the best current way to choose an architecture.

We learned how to use the checkpoint callback in Keras to give us the power to go back in time and find a version of the model that has performance characteristics we like. Then we created and used a custom callback to measure ROC AUC score as the model trained. We wrapped up by looking at how to use the Keras .predict() method with traditional metrics from sklearn.metrics.

In the next chapter, we'll take a look at multiclass classification, and we will talk more about how to prevent over fitting in the process.