Chapter 7: Implementing NLP Applications

In Chapter 6, Recurrent Neural Networks for Demand Prediction, we introduced Recurrent Neural Networks (RNNs) as a family of neural networks that are especially powerful to analyze sequential data. As a case study, we trained a Long Short-Term Memory (LSTM)-based RNN to predict the next value in the time series of consumed electrical energy. However, RNNs are not just suitable for strictly numeric time series, as they have also been applied successfully to other types of time series.

Another field where RNNs are state of the art is Natural Language Processing (NLP). Indeed, RNNs have been applied successfully to text classification, language models, and neural machine translation. In all of these tasks, the time series is a sequence of words or characters, rather than numbers.

In this chapter, we will run a short review of some classic NLP case studies and their RNN-based solutions: a sentiment analysis application, a solution for free...

Exploring Text Encoding Techniques for Neural Networks

In Chapter 4, Building and Training a Feedforward Neural Network, you learned that feedforward networks – and all other neural networks as well – are trained on numbers and don't understand nominal values. In this chapter, we want to feed words and characters into neural networks. Therefore, we need to introduce some techniques to encode sequences of words or characters – that is, sequences of nominal values – into sequences of numbers or numerical vectors. In addition, in NLP applications with RNNs, it is mandatory that the order of words or characters in the sequence is retained throughout the text encoding procedure.

Let's have a look at some text encoding techniques before we dive into the NLP case studies.

Index Encoding

In Chapter 4, Building and Training a Feedforward Neural Network, you learned about index encoding for nominal values. The idea was to represent each nominal...

Finding the Tone of Your Customers' Voice – Sentiment Analysis

A common use case for NLP is sentiment analysis. Here, the goal is to identify the underlying emotion in some text, whether positive or negative, and all the nuances in between. Sentiment analysis is implemented in many fields, such as to analyze incoming messages, emails, reviews, recorded conversations, and other similar texts.

Generally, sentiment analysis belongs to a bigger group of NLP applications known as text classification. In the case of sentiment analysis, the goal is to predict the sentiment class.

Another common example of text classification is language detection. Here, the goal is to recognize the text language. In both cases, if we use an RNN for the task, we need to adopt a many-to-one architecture. A many-to-one neural architecture accepts a sequence of inputs at different times, , and uses the final state of the output unit to predict the one single class – that is, sentiment...

Generating Free Text with RNNs

Now that we have seen how RNNs can be used for text classification, we can move on to the next case study. Here, we want to train an RNN to generate new free text in a certain style, be it Shakespearean English, a rap song, or mimicking a Brothers Grimm fairy tale. We will focus on the last application: training a network to generate free text in the style of Brothers Grimm fairy tales. However, the network and the process can be easily adjusted to produce a new rap song or a text in old Shakespearean English.

So, how can we train an RNN to generate new text?

The Dataset

First of all, you need a text corpus to train the network to generate new text. Any text corpus is good. However, keep in mind that the text you use for training will define the style of the text automatically generated. If you train the network on Shakespearean theater, you will get new text in old Shakespearean English; if you train the network on rap songs, you will get urban...

Generating Product Names with RNNs

This last NLP case study is similar to the previous one. There, we wanted the network to create new free text based on a start sequence; here, we want the network to create new free words based on a start token. There, we wanted the network to create new sequences of words; here, we want the network to create new sequences of characters. Indeed, the goal of this product name generation case study is to create new names – that is, new words. While there'll be some differences, the approaches will be similar.

In this section, we will explore the details of this new approach.

The Problem of Product Name Generation

Normally, we don't associate artificial intelligence with creativity, as it is usually used to predict the outcome based on previously seen examples. The challenge for this case study is to use artificial intelligence to create something new, which is thought to be in the domain of creative minds.

Let's take...

Defining and Training the Network Architecture

The process of designing and training the network is similar to the process used in the previous NLP case studies.

Designing the Network

In this case, we want to use a network with five layers:

• A Keras input layer to define the input shape
• A Keras LSTM layer for the sequence analysis
• A Keras dropout layer for regularization
• A Keras dense layers with linear activation
• A Keras softmax layer to transform the output into a probability distribution

The number of unique characters in the training set – that is, the character set size – is 95. Since we allow sequences of variable length, the shape of the input layer is ?, 95. The ? stands for a variable sequence length.

Next, we have the Keras LSTM Layer node. This time, it is important to activate the Return sequences and Return state checkboxes, as we need the intermediate output states during the training process and the cell state in...

Summary

We have reached the end of this relatively long chapter. Here, we have described three NLP case studies, each one solved by training an LSTM-based RNN applied to a time series prediction kind of problem.

The first case study analyzed movie review texts to extract the sentiment hidden in it. We dealt there with a simplified problem, considering a binary classification (positive versus negative) rather than considering too many nuances of possible user sentiment.

The second case study was language modeling. Training an RNN on a given text corpus with a given style produced a network capable of generating free text in that given style. Depending on the text corpus on which the network is trained, it can produce fairy tales, Shakespearean dialogue, or even rap songs. We showed an example that generates text in fairy tale style. The same workflows can be easily extended with more success to generate rap songs (R. Silipo, AI generated rap songs, CustomerThink, 2019, https:...

Questions and Exercises

1. What is a word embedding?

   a) An encoding functionality that can be trained within the neural network

   b) A text cleaning procedure

   c) A training algorithm for an RNN

   d) A postprocessing technique to choose the most likely character

2. Which statement regarding sentiment analysis is true?

   a) Sentiment analysis can only be solved with RNNs.

   b) Sentiment analysis is the same as emotion detection.

   c) Sentiment analysis identifies the underlying sentiment in a text.

   d) Sentiment analysis is an image processing task.

3. What does a many-to-many architecture mean?

   a) An architecture with an input sequence and an output sequence

   b) An architecture with an input sequence and a vector as output

   c) An architecture with many hidden units and many outputs

   d) An architecture with one input feature and an output sequence

4. Why do I need a trigger sequence for free text generation?

   a) To calculate the probabilities

   b) To compare the prediction with the target

   c) To initialize the...