Chapter 14. Deep Reinforcement Learning in NLP

Reinforcement learning in natural language processing (NLP) became a hot topic of research in the artificial intelligence community no more than a year ago. Most of the research publications catering to the use of reinforcement learning in NLP were published in the latter half of 2017.

The biggest reason behind the use of a reinforcement learning framework in any domain is the representation of the environment in the form of state, an exhaustive list of all possible actions in the environment, and a domain-specific reward function to achieve the goal through the most optimized path of actions. Thus, if a system has many possible actions but the correct set of actions is not given, and the objective highly depends on different options (actions) of the system then reinforcement learning framework can model the system better than existing supervised or unsupervised models.

Why use reinforcement learning in NLP ?

• NLP-oriented systems, such as text...

Text summarization

Text summarization is the process of automatically generating summarized text of the document test fed as an input by retaining the important information of the document. Text summarization condenses a big set of information in a concise manner; therefore, summaries play an important role in applications related to news/articles, text search, and report generation.

There are two types of summarization algorithms:

• Extractive summarization: Creates summaries by copying parts of the text from the input text
• Abstractive summarization: Generates new text by rephrasing the text or using new words that were not in the input text

The attention-based encoder decoder model created for machine translation (Bahdanau et al., 2014)is a sequence-to-sequence model and was able to generate abstractive summaries with good performance by achieving good ROUGE score (seeAppendix A, Further topics in Reinforcement Learning). The performance was good on short input sequences and it deteriorated...

Text question answering

Question answering is the task where a document context is provided along with a question whose answer is present within the given document context. Existing models for question answering used to optimize the cross-entropy loss, which used to encourage the exact answers and penalize other probable answers that are equally accurate as the exact answer. These existing question answering models (state of the art dynamic coattention network by Xiong et. al. 2017) are trained to output exact answer spans from the document context for the question asked. The start and end position of the actual ground truth answer is used as the target for this supervised learning approach. Thus, this supervised model uses cross-entropy loss over both the positions and the objective is to minimize this overall loss over both the positions.

As we can see, the optimization is done by using the positions and evaluation is done by using the textual content of the answer. Thus, there is a disconnect...

Summary

In this chapter, we learned how reinforcement learning can disrupt the domain of NLP. We studied the reasons behind the use of reinforcement learning in NLP. We covered two big application domains in NLP, that is, text summarization and question answering, and understood the basics of how a reinforcement learning framework was implemented in the existing models to obtain state-of-the-art results. There are other application domains in NLP where reinforcement learning has been implemented, such as dialog generation and machine translation (discussing them is out of the scope of this book).

This brings us to the end of this amazing journey of deep reinforcement learning. We started with the basics by understanding the concepts, then implemented those concepts using TensorFlow and OpenAI Gym, and went through cool research areas where deep reinforcement learning is being implemented at the core level. I hope the journey was interesting and we were able to build the best foundation possible...