Deep Networks for Text Classification

Text data belongs to the unstructured category of data. When developing deep network models, we need to complete additional preprocessing steps due to the unique nature of such data. In this chapter, you will learn about the steps you'll need to follow to develop text classification models using deep neural networks. This process will be illustrated with easy– to– follow examples. Text data, such as customer comments, product reviews, and movie reviews, plays an important role in businesses, and text classification is an important deep learning problem.

In this chapter, we will discuss two text datasets, learn how to prepare text data when developing deep network classification models, look at IMDb movie review data, develop a deep network architecture, fit and evaluate the model, and discuss some tips and best practices...

Text datasets

Text data can be used when we want to practice developing deep network models. Such data can be obtained from several publicly available sources. We will go over two such resources in this section:

• The UCI machine learning repository
• Text data within Keras

The UCI machine learning repository

The following link provides a variety of datasets that contain text sentences that have been extracted from reviews of products (from amazon.com), reviews of movies (from IMDB.com), and reviews of restaurants (from yelp.com): https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences.

Each sentence is labeled in terms of the sentiment that was expressed in the reviews. This sentiment is either positive or negative...

Preparing the data for model building

The steps we need to follow in order to prepare the data for model building are as follows:

1. Tokenization
2. Converting text into integers
3. Padding and truncation

To illustrate the steps involved in data preparation, we will make use of a very small text dataset involving five tweets related to when the Apple iPhone X released in September 2017. We will use this small dataset to understand the steps that are involved in data preparation and then we will switch to a larger IMDb dataset in order to build a deep network classification model. The following are the five tweets that we are going to store in t1 to t5:

t1 <- "I'm not a huge $AAPL fan but $160 stock closes down $0.60 for the day on huge volume isn't really bearish"
t2 <- "$AAPL $BAC not sure what more dissapointing: the new iphones or the presentation for...

Developing deep neural networks

Although we won't be developing a classification model based on just five tweets, let's look at the code for our model's architecture:

model <- keras_model_sequential()
model %>% layer_embedding(input_dim = 10, 
                          output_dim = 8, 
                          input_length = 5) 
summary(model)

OUTPUT
__________________________________________________________________________________
Layer (type)                        Output Shape                     Param #      
==================================================================================
embedding_1 (Embedding)             (None, 5, 8)                     80           
==================================================================================
Total params: 80
Trainable params: 80
Non-trainable params: 0
________________________________________________________________________________...

Model evaluation and prediction

Now, we will evaluate the model using training and test data to obtain the loss, accuracy, and confusion matrices. Our objective is to obtain a model that can classify sentiment contained in movie reviews as either positive or negative.

Evaluation using training data

The code to obtain the loss and accuracy values from the training data is as follows:

model %>% evaluate(train_x, train_y)
$loss
[1] 0.3745659
$acc
[1] 0.83428

As we can see, for training data, the loss and accuracy are 0.375 and 0.834, respectively. To look deeper into the model's sentiment classification performance, we need to develop a confusion matrix. To do so, use the following code:

pred <- model %>%   predict_classes...

Performance optimization tips and best practices

Now that we've obtained the test data's movie review classification accuracy, that is, 79%, we can work on improving this accuracy even further. Arriving at such an improvement may involve experimenting with the parameters in the model's architecture, the parameters that were used when we compiled the model, and/or the settings that were used while we were fitting a model. In this section, we will carry out an experiment by changing the maximum length of the sequence of words and, at the same time, use a different optimizer compared to what we used in the previous model.

Experimenting with the maximum sequence length and the optimizer

Let's start by creating...

Summary

In this chapter, we started by developing deep neural networks for text classification. Due to the unique characteristics of text data, several extra preprocessing steps are required before a deep neural network sentiment classification model can be developed. We used a small sample of five tweets to go over the preprocessing steps, including tokenization, converting text data into a sequence of integers, and padding/truncation to arrive at the same sequence length. We also highlighted that automatically labeling text sequences with the appropriate sentiment is a challenging problem and general lexicons may be unable to provide useful results.

To develop a deep network sentiment classification model, we switched to a larger and ready-to-use IMDb movie review dataset that's available as part of Keras. To optimize the model's performance, we also experimented with...