Natural Language Processing Using Deep Learning

This chapter will demonstrate how to use deep learning for natural language processing (NLP). NLP is the processing of human language text. NLP is a broad term for a number of different tasks involving text data, which include (but are not limited to) the following:

• Document classification: Classifying documents into different categories based on their subject
• Named entity recognition: Extracting key information from documents, for example, people, organizations, and locations
• Sentiment analysis: Classifying comments, tweets, or reviews as positive or negative sentiment
• Language translation: Translating text data from one language to another
• Part of speech tagging: Assigning the type to each word in a document, which is usually used in conjunction with another task

In this chapter, we will look at document classification, which...

Document classification

This chapter will be looking at text classification using Keras. The dataset we will use is included in the Keras library. As we have done in previous chapters, we will use traditional machine learning techniques to create a benchmark before applying a deep learning algorithm. The reason for this is to show how deep learning models perform against other techniques.

The Reuters dataset

We will use the Reuters dataset, which can be accessed through a function in the Keras library. This dataset has 11,228 records with 46 categories. To see more information about this dataset, run the following code:

library(keras)
?dataset_reuters

Although the Reuters dataset can be accessed from Keras, it is not in a format...

Advanced deep learning text classification

Our basic deep learning model is much less complex than the traditional machine learning approach, but its performance is not quite as good. This section looks at some advanced techniques for text classification in deep learning. The following sections explain a number of different approaches and focus on code examples rather than heavy deep explanations. If you are interested in more detail, then look at the book Deep Learning by Goodfellow, Bengio, and Courville (Goodfellow, Ian, et al. Deep learning. Vol. 1. Cambridge: MIT Press, 2016.). Another good reference that covers NLP in deep learning is a book by Yoav Goldberg (Goldberg, Yoav. Neural network methods for natural language processing).

1D convolutional neural network model

...

Summary

We really covered a lot in this chapter! We built a fairly complex traditional NLP example that had many hyperparameters, as well as training it on several machine learning algorithms. It achieved a reputable result of getting 95.24% accuracy. However, when we looked into traditional NLP in more detail, we found that it had some major problems: it requires non-trivial feature engineering, it creates sparse high-dimensional data frames, and it may require discarding a substantial amount of data before machine learning.

In comparison, the deep learning approach uses word vectors or embeddings, which are much more efficient and do not require preprocessing. We ran through a number of deep learning approaches, including 1D convolutional layers, Recurrent Neural Networks, GRUs, and LSTM. We finally combined the two best previous approaches into one approach in our final model...