Approaches to Natural Language Understanding – Rule-Based Systems, Machine Learning, and Deep Learning

This chapter will review the most common approaches to natural language understanding (NLU) and discuss both the benefits and drawbacks of each approach, including rule-based techniques, statistical techniques, and deep learning. It will also discuss popular pre-trained models such as Bidirectional Encoder Representations from Transformers (BERT) and its variants. We will learn that NLU is not a single technology; it includes a range of techniques, which are applicable to different goals.

In this chapter, we cover the following main topics:

• Rule-based approaches
• Traditional machine-learning approaches
• Deep learning approaches
• Pre-trained models
• Considerations for selecting technologies

Let’s begin!

Rule-based approaches

The basic idea behind rule-based approaches is that language obeys rules about how words are related to their meanings. For example, when we learn foreign languages, we typically learn specific rules about what words mean, how they’re ordered in sentences, and how prefixes and suffixes change the meanings of words. The rule-based approach to NLU operates on the premise that these kinds of rules can be provided to an NLU system so that the system can determine the meanings of sentences in the same way that a person does.

The rule-based approach was widely used in NLU from the mid-1950s through the mid-1990s until machine-learning-based approaches became popular. However, there are still NLU problems where rule-based approaches are useful, either on their own or when combined with other techniques.

We will begin by reviewing the rules and data that are relevant to various aspects of language.

Words and lexicons

Nearly everyone is familiar with...

Traditional machine learning approaches

While rule-based approaches provide very fine-grained and specific information about language, there are some drawbacks to these approaches, which has motivated the development of alternatives. There are two major drawbacks:

• Developing the rules used in rule-based approaches can be a laborious process. Rule development can either be done by experts directly writing rules based on their knowledge of the language or, more commonly, the rules can be derived from examples of text that have been annotated with a correct analysis. Both of these approaches can be expensive and time-consuming.
• Rules are not likely to be universally applicable to every text that the system encounters. The experts who developed the rules might have overlooked some cases, the annotated data might not have examples of every case, and speakers can make errors such as false starts, which need to be analyzed although they aren’t covered by any rule. Written...

Deep learning approaches

Neural networks, and especially the large neural networks generally referred to as deep learning, have become very popular for NLU in the past few years because they significantly improve the accuracy of earlier methods.

The basic concept behind neural networks is that they consist of layers of connected units, called neurons in analogy to the neurons in animal nervous systems. Each neuron in a neural net is connected to other neurons in the neural net. If a neuron receives the appropriate inputs from other neurons, it will fire, or send input to another neuron, which will in turn fire or not fire depending on other inputs that it receives. During the training process, weights on the neurons are adjusted to maximize classification accuracy.

Figure 3.5 shows an example of a four-layer neural net performing a sentiment analysis task. The neurons are circles connected by lines. The first layer, on the left, receives a text input. Two hidden layers of neurons...

Pre-trained models

The most recent approach to NLU is based on the idea that much of the information required to understand natural language can be made available to many different applications by processing generic text (such as internet text) to create a baseline model for the language. Some of these models are very large and are based on tremendous amounts of data. To apply these models to a specific application, the generic model is adapted to the application through the use of application-specific training data, through a process called fine-tuning. Because the baseline model already contains a vast amount of general information about the language, the amount of training data can be considerably less than the training data required for some of the traditional approaches. These popular technologies include BERT and its many variations and Generative Pre-trained Transformers (GPTs) and their variations.

Pre-trained models will be discussed in detail in Chapter 11.

Considerations for selecting technologies

This chapter has introduced four classes of NLU technologies:

• Rule-based
• Statistical machine learning
• Deep learning and neural networks
• Pre-trained models

How should we decide which technology or technologies should be employed to solve a specific problem? The considerations are largely practical and have to do with the costs and effort required to create a working solution. Let’s look at the characteristics of each approach.

Table 3.3 lists the four approaches to NLU that we’ve reviewed in this chapter and how they compare with respect to developer expertise, the amount of data required, the training time, accuracy, and cost. As Table 3.3 shows, every approach has advantages and disadvantages. For small or simple problems that don’t require large amounts of data, the rule-based, deep learning, or pre-trained approaches should be strongly considered, at least for part of the pipeline. While...

Summary

In this chapter, we surveyed the various techniques that can be used in NLU applications and learned several important skills.

We learned about what rule-based approaches are and the major rule-based techniques, including topics such as POS tagging and parsing. We then learned about the important traditional machine learning techniques, especially the ways that text documents can be represented numerically. Next, we focused on the benefits and drawbacks of the more modern deep learning techniques and the advantages of pre-trained models.

In the next chapter, we will review the basics of getting started with NLU – installing Python, using Jupyter Labs and GitHub, using NLU libraries such as NLTK and spaCy, and how to choose between libraries.