Identifying Practical Natural Language Understanding Problems

In this chapter, you will learn how to identify natural language understanding (NLU) problems that are a good fit for today’s technology. That means they will not be too difficult for the state-of-the-art NLU approaches but neither can they be addressed by simple, non-NLU approaches. Practical NLU problems also require sufficient training data. Without sufficient training data, the resulting NLU system will perform poorly. The benefits of an NLU system also must justify its development and maintenance costs. While many of these considerations are things that project managers should think about, they also apply to students who are looking for class projects or thesis topics.

Before starting a project that involves NLU, the first question to ask is whether the goals of the project are a good fit for the current state of the art in NLU. Is NLU the right technology for solving the problem that you wish to address?...

Identifying problems that are the appropriate level of difficulty for the technology

Note

This chapter is focused on technical considerations. Questions such as whether a market exists for a proposed application, or how to decide whether customers will find it appealing, are important questions, but they are outside of the scope of this book.

Here are some kinds of problems that are a good fit for the state of the art.

Today’s NLU is very good at handling problems based on specific, concrete topics, such as these examples:

• Classifying customers’ product reviews into positive and negative reviews: Online sellers typically offer buyers a chance to review products they have bought, which is helpful for other prospective buyers as well as for sellers. But large online retailers with thousands of products are then faced with the problem of what to do with the information from thousands of reviews. It’s impossible for human tabulators to read all the...

Taking development costs into account

After making sure that data is available, and that the data is (or can be) annotated with the required intents, entities, and classification categories, the next consideration for deciding whether NLP is a good fit for an application is the cost of developing the application itself. Some technically feasible applications can nevertheless be impractical because they would be too costly, risky, or time-consuming to develop.

Development costs include determining the most effective machine learning approaches to a specific problem. This can take significant time and involve some trial and error as models need to be trained and retrained in the process of exploring different algorithms. Identifying the most promising algorithms is also likely to require NLP data scientists, who may be in short supply. Developers have to ask the question of whether the cost of development is consistent with the benefits that will be realized by the final application...

Taking maintenance costs into account

The final consideration for natural language applications, especially deployed applications, is the cost of maintenance. This is easy to overlook because NLU applications have several maintenance considerations that don’t apply to most traditional applications. Specifically, the type of language used in some applications changes over time. This is expected since it reflects changes in the things that the users are talking about. In customer service applications, for example, product names, store locations, and services change, sometimes very quickly. The new vocabulary that customers use to ask about this information changes as well. This means that new words have to be added to the system, and machine learning models have to be retrained.

Similarly, applications that provide rapidly changing information need to be kept up to date on an ongoing basis. As an example, the word COVID-19 was introduced in early 2020 – no one had ever...

A flowchart for deciding on NLU applications

This chapter has covered many considerations that should be taken into account in deciding on an NLP application.

Figure 2.2 summarizes these considerations as a flowchart of the process for evaluating a potential NLU application.

Figure 2.2 – Steps in evaluating an NLU project

Starting at the top, the process starts by asking whether the problem is too hard or too easy for the current state of the art, using the criteria discussed earlier. If it’s either too hard or too easy, we should look for another application, or look at cutting back or expanding the scope of the application to make it a better fit for NLP technology. For example, the application might be redesigned to handle fewer languages.

If the problem seems to be a good fit for the state of the art, the next steps are to ensure that the appropriate data is available, and if not, whether data can be collected. Once data is available...

Summary

In this chapter, we covered the topic of selecting NLP applications that have a good chance of success with current NLP technology. Successful applications generally have input with specific, objective answers, have training data available, and handle (at most) a few languages.

Specifically, this chapter addressed a number of important questions. We learned how to identify problems that are the appropriate level of difficulty for the current state of the art of NLU technology. We also learned how to ensure that sufficient data is available for system development and how to estimate the costs of development and maintenance.

Learning how to evaluate the feasibility of different types of NLP applications as discussed in this chapter will be extremely valuable as you move forward with your NLP projects. Selecting an application that is too ambitious will result in frustration and a failed project, whereas selecting an application that is too easy for the state of the art...