Chapter 3. Analyzing Videos

In the previous chapter, we looked into different APIs for processing images. In this chapter, we will cover two new APIs: the Emotion API and the Video API. Working our way through this chapter, we will add emotion recognition to our smart-house application. We will also learn how to detect emotions in videos. Additionally, we will learn how to use the different operations of the Video API.

In this chapter, we will cover the following topics:

• Recognizing emotions in images

• Retrieving an aggregated summary of emotion from videos

• Stabilizing and smoothing shaky videos

• Detecting and tracking faces in videos

• Detecting motion in videos with a stationary background

• Automatically generating video thumbnail summaries

Knowing your mood using the Emotion API

The Emotion API allows you to recognize emotions from faces, both in images and videos.

Research has shown that there are some key emotions that can be classified as cross-cultural. These are happiness sadness, surprise, anger, fear, contempt, disgust, and neutral. All of these are detected by the API, which allows your applications to respond in a more personalized way, by knowing the user's mood.

We will learn how to recognize emotions from images first, so our smart-house application can know our mood.

Getting images from a web camera

Imagine that there are several cameras around your house. The smart-house application can see how your mood is at any time. By knowing this, it can utilize the mood to better predict your needs.

We are going to add web-camera capabilities to our application. In case you do not have a web camera, you can follow along, but load images using the techniques we have already seen.

First we need to add a NuGet package to our smart...

Diving into the Video API

The Video API provides advanced algorithms for video processing. Using this API, you can build more personalized and intelligent applications.

There are four main features of this API, which we will look into. They are:

• Face detection and tracking

• Motion detection

• Stabilizing shaky videos

• Generating intelligent video thumbnails

Note

If you have not already done so, register for an API key for the Video API at https://www.microsoft.com/cognitive-services/en-us/video-api.

Microsoft have created a NuGet package, containing the Video API, which allows us to utilize all the features available. Let us start by creating a new project in Visual Studio, based off our MVVM template.

In the newly created project, add Microsoft.ProjectOxford.Video through the NuGet Package Manager.

As this example project will have a lot of common functionality, we just need one View. The example code provided has added a View, VideoView.xaml, to the View folder, and a corresponding view model. You...

Analyzing emotions in videos

Earlier, we looked at analyzing emotions in images. We can do the same analysis with videos as well.

To be able to do this, we can modify the existing example for the Video API.

Start by adding Microsoft.ProjectOxford.Emotion as a NuGet client package.

Next we add Emotion to the AvailableOperations enum. In the VideoOperations class, add a new case for this value, in CreateVideoOperationSettings. Return null, as we do not need any video-operation settings for this.

Add a private member to VideoOperations:

    private EmotionServiceClient _emotionServiceClient; 

Initialize this in the constructor, using the API key you registered earlier.

In the VideoOperationResultEventArgs, add a new property called EmotionResult. This should be of the type VideoAggregateRecognitionResult.

Back in the VideoOperations class, copy the GetVideoOperationResultAsync function. Rename it to GetVideoEmotionResultAsync, and change the accepted parameter to VideoEmotionRecognitionOperation...

Summary

Throughout this chapter, we have covered the Emotion API and the Video API. We started by making the smart-house application see what kind of mood you are in. Following this, we dived deep into the Video API, where we learned how to detect and track faces, detect motion, stabilize videos, and generate intelligent video thumbnails. To end the chapter, we moved back to the Emotion API, where we learned how to do emotion analysis on videos.

In the following chapter, we are moving away from the Vision APIs and into the first Language API. We will learn how to understand intent in sentences, using the power of the Language Understanding Intelligent Service (LUIS).