Chapter 6. Understanding Text

The previous chapter covered the Speech APIs. Throughout this chapter, we will look closer at more Language APIs. We will learn how to use spellcheck features. Moving on, we will see how to use web language models to generate and find the next word and sequences. We will then discover how to detect languages, key phrases, and sentiment in text. Finally, we will look briefly into linguistic analysis. Doing so will teach us how to explore the structure of text, discovering nouns, verbs, and more.

By the end of this chapter, we will have covered the following topics:

• Checking spelling and recognizing slang and informal language, common names, homonyms, and brands

• Breaking words, such as hashtags and URLs, into several words

• Finding the likelihood of a sequence of words appearing together, or a sequence appearing after a given word

• Detecting language, key phrases, and sentiment in text

• Executing linguistic analysis of text

Setting up a common core

Before we get into the details, we want to set ourselves up for success. At the time of writing, none of the Language APIs we will be covering have NuGet client packages. As such, we will need to call directly to the REST endpoints. Because of this, we will do some work beforehand, to make sure we get away with writing less code.

New project

We will not be adding the APIs to our Smart-House application. Go on and create a new project, using the MVVM template created in Chapter 1, Getting Started with Microsoft Cognitive Services:

1. Go into the NuGet package manager and install Newtonsoft.Json. This will help us deserialize API responses and serialize request bodies.

2. Right-click on References.

3. In the Assemblies tab, select System.Web and System.Runtime.Serialization.

4. Click OK.

5. In the MainView.xaml file, add a TabControl element. All our additional views will be added as TabItems in the MainView.

Web requests

All the APIs follow the same patterns. They call on their respective...

Correcting spelling errors

The Bing Spell Check API leverages the power of machine learning and statistical machine translation to train and evolve a highly contextual algorithm for spell-checking. Doing so allows us to utilize this to perform spell-checking using context.

A typical spell-checker will follow dictionary-based rule sets. As you can imagine, this will need continuous updates and expansions.

Using the Bing Spell Check API, we can recognize and correct slang and informal language. It can recognize common naming errors and correct word-breaking issues. It can detect and correct words that sound the same, but differ in meaning and spelling (homophones). It can also detect and correct brands and popular expressions.

Create a new View in the View folder. Call the file SpellCheckView.xaml. Add a TextBox element for the input query. We will also need two TextBox elements for pre- and post-context. Add a TextBox element to show the result, and a Button element to execute the spell check...

Natural Language Processing using the Web Language Model

Using the Web Language Model (Web LM) API, we are able to do natural language processing. The language models are trained on web-scale, using data collected by Bing.

This API offers four key features:

• Joint probability of a sequence of words

• Conditional probability of one word following a sequence of words

• Most likely words to follow a given sequence

• Word breaking of strings without any spaces

To get started, add a new file, called WebLmView.xaml, to the View folder. In this view, we want to have two TextBox elements, one for our input query and one for the result View. In addition, we want four Button elements. Each of these will trigger one of the API features.

Next, we need to add a new ViewModel. Add a new file, called WebLmViewModel.cs, to the ViewModel folder. Add two string properties, corresponding to the input and output in the View. We also need to add four ICommand objects, which will be triggered when the buttons are pressed.

To...

Extracting information through textual analysis

Using the Textual Analytics API, we are able to analyze text. We will cover language detection, key-phrase analysis, and sentiment analysis. In addition, a new feature is the ability to detect topics. This does, however, require a lot of sample text, and as such, we will not go into details on this.

For all our text analysis tasks, we will be using a new View. Add a new View into the View folder, called TextAnalysisView.xaml. This should contain a TextBox element for the input query. It should also have a TextBox element for the result. We will need three Button elements, one for each detection analysis we will perform.

We will also need a new ViewModel, so add TextAnalysisViewModel.cs to the ViewModel folder. In this we need two string properties, one for each TextBox. Also add three ICommand properties, one for each of our buttons.

Note

If you have not already done so, register for an API key at https://www.microsoft.com/cognitive-services/en...

Exploring text using linguistic analysis

Using the Linguistic Analysis API, we can explore the structure of text. We can access part-of-speech tagging and parsing, which allows us to tap into important concepts and actions. By using this API, we can understand the structure of a text. This can help us if we are mining customer feedback, interpreting user commands, or consuming web text.

As linguistic analysis is an advanced topic, we will not dive into all its details. Instead, we will go through a quick introduction to key concepts, and then see how we can do the analysis itself. We will not spend time on parsing the resulting data for the UI, so we will simply print the JSON result.

Note

For more information on linguistic analysis and natural language analysis, please refer to the following Wikipedia article: https://en.wikipedia.org/wiki/Natural_language_processing.

Introduction to linguistic analysis

At the time of writing, there are currently three available tools in the API. These are...

Summary

Throughout this chapter, we have focused on the Language APIs. We started by creating the parts needed to execute API calls to the different services. Following this, we looked at the Bing Spell Check API. Next, we learned how to use web language models to break words and calculate probability of word sequences. We moved into more analytical APIs, where we learned how to detect languages, key phrases, and sentiment. We completed the chapter by taking a brief look into linguistic analysis.

The next chapter will take us from Language APIs into Knowledge APIs. In this chapter, we will learn how to recognize and identify entities based on context. In addition, we will learn how to use the Recommendations API.