Learning Microsoft Cognitive Services,: Use Cognitive Services APIs to add AI capabilities to your applications, Third Edition

The best way to introduce Microsoft Cognitive Services is to see how it can be used in action. Microsoft (as well as other companies) has created a lot of example applications to show off its capabilities. Several may be seen as silly, such as the How-Old.net (http://how-old.net/) image analysis and the what if I were that person application. These applications have generated quite some buzz, and they show off some of the APIs in a good way.

The one demonstration that is truly inspiring, though, is the one featuring a visually impaired person. Talking computers inspired him to create an application to allow blind and visually impaired people to understand what is going on around them. The application has been built upon Microsoft Cognitive Services. It gives us a good idea of how these APIs can be used to change the world, for the better. Before moving on, head over to https://www.youtube.com/watch?v=R2mC-NUAmMk and take a peek into the world of Microsoft Cognitive Services.

Before we start diving into the action, we will go through some initial setup. More to the point, we will set up some boilerplate code that we will utilize throughout this book.

To get started, you will need to install a version of Visual Studio, preferably Visual Studio 2015 or later. The Community Edition will work fine for this purpose. You do not need anything more than what the default installation offers.

Throughout this book, we will utilize the different APIs to build a smart-house application. The application will be created to see how a futuristic house might appear. If you have seen the Iron Man movies, you can think of the application as resembling Jarvis, in some ways.

In addition, we will be making smaller sample applications using the Cognitive Services APIs. Doing so will allow us to look at each API, even those that did not make it to the final application.

What's common with all the applications that we will build is that they will be Windows Presentation Foundation (WPF) applications. This is fairly well known, and allows us to build applications using the Model-View-ViewModel (MVVM) pattern. One of the advantages of taking this road is that we will be able to see the API usage quite clearly. It also separates code so that you can bring the API logic to other applications with ease.

The following steps describe the process of creating a new WPF project:

We will not go through the MVVM pattern in detail, as this is out of the scope of this book. The key takeaway from the screenshot is that we have separated the View from what becomes the logic. We then rely on the ViewModel to connect the pieces.

To be able to run this, however, we do need to set up our project. Go through the following steps:

Following this, we need to fill in the content of the ObservableObject.cs file. We start off by having it inherit from the INotifyPropertyChanged class as follows:

        public class ObservableObject : INotifyPropertyChanged

This is a rather small class, which should contain the following:

        public event PropertyChangedEventHandlerPropertyChanged;
        protected void RaisePropertyChangedEvent(string propertyName)
        {
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
        }

We declare a property changed event and create a function to raise the event. This will allow the user interface (UI) to update its values when a given property has changed.

We also need to be able to execute actions when buttons are clicked. This can be achieved when we put some content into the DelegateCommand.cs file. Start by making the class inherit the ICommand class, and declare the following two variables:

        public class DelegateCommand : ICommand
        {
            private readonly Predicate<object> _canExecute;
            private readonly Action<object> _execute;

The two variables we have created will be set in the constructor. As you will notice, you are not required to add the _canExecute parameter, and you will see why in a bit:

            public DelegateCommand(Action<object> execute, Predicate<object> canExecute = null)
            {
                _execute = execute;
                _canExecute = canExecute;
            }

To complete the class, we add two public functions and one public event, as follows:

        public bool CanExecute(object parameter)
        {
            if (_canExecute == null) return true;
            return _canExecute(parameter);
        }

        public void Execute(object parameter)
        {
            _execute(parameter);
        }
  
        public event EventHandlerCanExecuteChanged
        {
            add
            {
                CommandManager.RequerySuggested += value;
            }
            remove
            {
                CommandManager.RequerySuggested -= value;
            }
        }
    }

The functions declared will return the corresponding predicate, or action, declared in the constructor. This will be something we declare in our ViewModel instances, which, in turn, will be something that executes an action or tells the application that it can or cannot execute an action. If a button is in a state where it is disabled (that is, when the CanExecute function returns false) and the state of the CanExecute function changes, the event that is declared will let the button know.

With that in place, you should be able to compile and run the application, so go on and try that. You will notice that the application does not actually do anything or present any data yet, but we have an excellent starting point.

Before we do anything else with the code, we are going to export the project as a template using the following steps. This is so that we do not have to redo all these steps for each small sample project we create:

By default, the template will be imported into Visual Studio again. We are going to test that it works immediately by creating a project for this chapter. So go ahead and create a new project, selecting the template that we just created. The template should be listed in the Visual C# section of the installed templates list. Call the project Chapter1, or something else, if you prefer. Make sure it compiles and that you are able to run it before we move to the next step.

With the newly created project, we will now try our first API, the Face API. We will not be doing a whole lot, but we will still see how simple it is to detect faces in images.

The steps we need to go through in order to do this are as follows:

Head over to https://portal.azure.com to start the process of registering for a free subscription to the Face API. You will be taken to a login page. Log on with your Microsoft account; if you do not have one, then register for one.

Once logged in, you will need to add a new resource by clicking on + New on the right-hand menu. Search for Face API and select the first entry, as shown in the following screenshot:

Enter a name and select the subscription, location, and pricing tier. At the time of writing, there are two pricing options, one free and one paid, as shown in the following screenshot:

Once created, you can go into the newly created resource. You will need one of the two available API keys. These can be found in the Keys option of the Resource Management menu, as shown in the following screenshot:

Some of the APIs that we will cover have their own NuGet packages created. Whenever this is the case, we will utilize those packages to do the operations we want to perform. A common feature of all APIs is that they are REST APIs, which means that in practice you can use them with whichever language you want. For those APIs that do not have their own NuGet package, we call the APIs directly through HTTP.

A NuGet package does exist for the Face API we are using now, so we need to add that to our project. Head over to the NuGet Package Manager option for the project we created earlier. In the Browse tab, search for the Microsoft.ProjectOxford.Face package and install the package from Microsoft, as shown in the following screenshot:

As you will notice, another package will also be installed. This is the Newtonsoft.Json package, which is required by the Face API.

The next step is to add a UI to our application. We will be adding this in the MainView.xaml file. Open this file where the template code that we created earlier should be. This means that we have DataContext, and can make bindings for our elements, which we will define now.

First, we add a grid and define some rows for the grid, as follows:

    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="*" />
            <RowDefinition Height="20" />
            <RowDefinition Height="30" />
        </Grid.RowDefinitions>

Three rows are defined. The first is a row where we will have an image, the second is a line for the status message, and the last is where we will place some buttons.

Next, we add our image element, as follows:

        <Image x:Name="FaceImage" Stretch="Uniform" Source=
            "{Binding ImageSource}" Grid.Row="0" />

We have given it a unique name. By setting the Stretch parameter to Uniform, we ensure that the image keeps its aspect ratio. Further on, we place this element in the first row. Last, we bind the image source to a BitmapImage in the ViewModel, which we will look at in a bit.

The next row will contain a text block with some status text. The Text property will be bound to a string property in the ViewModel, as follows:

        <TextBlockx:Name="StatusTextBlock" Text=
            "{Binding StatusText}" Grid.Row="1" />

The last row will contain one button to browse for an image and one button to be able to detect faces. The command properties of both buttons will be bound to the DelegateCommand properties in the ViewModel, as follows:

        <Button x:Name = "BrowseButton"
                  Content = "Browse" Height="20" Width="140" 
                  HorizontalAlignment = "Left"
                  Command="{Binding BrowseButtonCommand}"
                  Margin="5, 0, 0, 5"Grid.Row="2" />

        <Button x:Name="DetectFaceButton"
                  Content="Detect face" Height="20" Width="140"
                  HorizontalAlignment="Right"
                  Command="{Binding DetectFaceCommand}"
                  Margin="0, 0, 5, 5"Grid.Row="2"/>

With the View in place, make sure that the code compiles and runs it. This should present you with the following UI:

The last part of the process is to create the binding properties in our ViewModel and make the buttons execute something. Open the MainViewModel.cs file. The class should already inherit from the ObservableObject class. First, we define two variables as follows:

    private string _filePath;
    private IFaceServiceClient _faceServiceClient;

The string variable will hold the path to our image, while the IFaceServiceClient variable is to interface the Face API. Next, we define two properties, as follows:

    private BitmapImage _imageSource;
    public BitmapImageImageSource
    {
        get { return _imageSource; }
        set
        {
            _imageSource = value;
            RaisePropertyChangedEvent("ImageSource");
        }
    }

    private string _statusText;
    public string StatusText
    {
        get { return _statusText; }
        set
        {
           _statusText = value;
           RaisePropertyChangedEvent("StatusText");
        }
    }

What we have here is a property for the BitmapImage, mapped to the Image element in the View. We also have a string property for the status text, mapped to the text block element in the View. As you may also notice, when either of the properties is set, we call the RaisePropertyChangedEvent event. This will ensure that the UI updates when either property has new values.

Next, we define our two DelegateCommand objects and perform some initialization through the constructor, as follows:

    public ICommandBrowseButtonCommand { get; private set; }
    public ICommandDetectFaceCommand { get; private set; }

    public MainViewModel()
    {
        StatusText = "Status: Waiting for image...";

        _faceServiceClient = new FaceServiceClient("YOUR_API_KEY_HERE", "ROOT_URI);

        BrowseButtonCommand = new DelegateCommand(Browse);
        DetectFaceCommand = new DelegateCommand(DetectFace, CanDetectFace);
    }

The properties for the commands are public to get, but private to set. This means that we can only set them from within the ViewModel. In our constructor, we start off by setting the status text. Next, we create an object of the Face API, which needs to be created with the API key we got earlier. In addition, it needs to specify the root URI, pointing at the location of the service. It can, for instance, be https://westeurope.api.cognitive.microsoft.com/face/v1.0 if the service is located in west Europe.

If the service is located in the west US, you would replace westeurope with westus. The root URI can be found in the following place in Azure Portal:

At last, we create the DelegateCommand constructor for our command properties. Note how the browse command does not specify a predicate. This means that it will always be possible to click on the corresponding button. To make this compile, we need to create the functions specified in the DelegateCommand constructors: the Browse, DetectFace, and CanDetectFace functions.

We start the Browse function by creating an OpenFileDialog object. This dialog is assigned a filter for JPEG images, and, in turn, it is opened, as shown in the following code. When the dialog is closed, we check the result. If the dialog was canceled, we simply stop further execution:

    private void Browse(object obj)
    {
        var openDialog = new Microsoft.Win32.OpenFileDialog();
        openDialog.Filter = "JPEG Image(*.jpg)|*.jpg";
        bool? result = openDialog.ShowDialog();

        if (!(bool)result) return;

With the dialog closed, we grab the filename of the file selected and create a new URI from it, as shown in the following code:

        _filePath = openDialog.FileName;
        Uri fileUri = new Uri(_filePath);

With the newly created URI, we want to create a new BitmapImage. We specify it so that it uses no cache, and we set the URI source of the URI that we created, as shown in the following code:

        BitmapImage image = new BitmapImage(fileUri);

        image.CacheOption = BitmapCacheOption.None;
        image.UriSource = fileUri;

The last step we take is to assign the bitmap image to our BitmapImage property so that the image is shown in the UI. We also update the status text to let the user know that the image has been loaded, as shown in the following code:

        ImageSource = image;
        StatusText = "Status: Image loaded...";
    }

The CanDetectFace function checks whether or not the DetectFacesButton button should be enabled. In this case, it checks whether our image property actually has a URI. If it does by extension, then that means that we have an image and we should be able to detect faces, as shown in the following code:

    private boolCanDetectFace(object obj)
    {
        return !string.IsNullOrEmpty(ImageSource?.UriSource.ToString());
    }

Our DetectFace method calls an async method to upload and detect faces. The return value contains an array of the FaceRectangles variable. This array contains the rectangle area for all face positions in the given image. We will look into the function that we are going to call in a bit.

After the call has finished executing, we print a line containing the number of faces to the debug console window, as follows:

    private async void DetectFace(object obj)
    {
        FaceRectangle[] faceRects = await UploadAndDetectFacesAsync();

        string textToSpeak = "No faces detected";

        if (faceRects.Length == 1)
            textToSpeak = "1 face detected";
        else if (faceRects.Length> 1)
            textToSpeak = $"{faceRects.Length} faces detected";

        Debug.WriteLine(textToSpeak);
    }

In the UploadAndDetectFacesAsync function, we create a Stream from the image, as shown in the following code. This stream will be used as input for the actual call to the Face API service:

    private async Task<FaceRectangle[]>UploadAndDetectFacesAsync()
    {
        StatusText = "Status: Detecting faces...";

        try
        {
            using (Stream imageFileStream = File.OpenRead(_filePath))

The following line is the actual call to the detection endpoint for the Face API:

            Face[] faces = await _faceServiceClient.DetectAsync(imageFileStream, true, true, new List<FaceAttributeType>() { FaceAttributeType.Age });

The first parameter is the file stream that we created in the previous step. The rest of the parameters are all optional. The second parameter should be true if you want to get a face ID. The next parameter specifies whether you want to receive face landmarks or not. The last parameter takes a list of facial attributes that you may want to receive. In our case, we want the age parameter to be returned, so we need to specify that.

The return type of this function call is an array of faces, with all the parameters that you have specified, as shown in the following code:

            List<double> ages = faces.Select(face =>face.FaceAttributes.Age).ToList();
            FaceRectangle[] faceRects = faces.Select(face =>face.FaceRectangle).ToArray();

            StatusText = "Status: Finished detecting faces...";

            foreach(var age in ages) {
                Console.WriteLine(age);
            }
            return faceRects;
        }
    }

The first line iterates over all faces and retrieves the approximate age of all faces. This is later printed to the debug console window, in the foreach loop.

The second line iterates over all faces and retrieves the face rectangle, with the rectangular location of all faces. This is the data that we return to the calling function.

Add a catch clause to finish the method. Where an exception is thrown in our API call, we catch that. We want to show the error message and return an empty FaceRectangle array.

With that code in place, you should now be able to run the full example. The end result will look like the following screenshot:

The result debug console window will print the following text:

    1 face detected
    23,7

Now that you have seen a basic example of how to detect faces, it is time to learn a bit about what else Cognitive Services can do for you. When using Cognitive Services, you have 21 different APIs to hand. These are, in turn, separated into five top-level domains depending on what they do. These domains are vision, speech, language, knowledge, and search. We will learn more about them in the following sections.

Now that we have seen what else Microsoft Cognitive Services can offer, we are going to add an API to our face detection application. In this section, we will add the Bing Speech API to make the application say the number of faces out loud.

This feature of the API is not provided in the NuGet package, and as such, we are going to use the REST API.

To reach our end goal, we are going to add two new classes, TextToSpeak and Authentication. The first class will be in charge of generating the correct headers and making the calls to our service endpoint. The latter class will be in charge of generating an authentication token. This will be tied together in our ViewModel, where we will make the application speak back to us.

We need to get our hands on an API key first. Head over to the Microsoft Azure Portal. Create a new service for Bing Speech.

To be able to call the Bing Speech API, we need to have an authorization token. Go back to Visual Studio and create a new file called Authentication.cs. Place this in the Model folder.

We need to add two new references to the project. Find the System.Runtime.Serialization and System.Web packages in the Assembly tab in the Add References window and add them.

In our Authentication class, define four private variables and one public property, as follows:

    private string _requestDetails;
    private string _token;
    private Timer _tokenRenewer;

    private const int TokenRefreshInterval = 9;

    public string Token { get { return _token; } }

The constructor should accept one string parameter, clientSecret. The clientSecret parameter is the API key you signed up for.

In the constructor, assign the _clientSecret variable, as follows:

    _clientSecret = clientSecret;

Create a new function called Initialize, as follows:

    public async Task Initialize()
    {
        _token = GetToken();

        _tokenRenewer = new Timer(new TimerCallback(OnTokenExpiredCallback), this,
        TimeSpan.FromMinutes(TokenRefreshInterval),
        TimeSpan.FromMilliseconds(-1));
    }

We then fetch the access token in a method that we will create shortly.

Finally, we create our timer class, which will call the callback function in nine minutes. The callback function will need to fetch the access token again and assign it to the _token variable. It also needs to ensure that we run the timer again in nine minutes.

Next, we need to create the GetToken method. This method should return a Task<string>object, and it should be declared as private and marked as async.

In the method, we start by creating an HttpClient object, pointing to an endpoint that will generate our token. We specify the root endpoint and add the token issue path, as follows:

    using(var client = new HttpClient())
    {
        client.DefaultRequestHeaders.Add ("Opc-Apim-Subscription-Key", _clientSecret);
        UriBuilder uriBuilder = new UriBuilder (https://api.cognitive.microsoft.com/sts/v1.0");
        uriBuilder.Path = "/issueToken";

We then go on to make a POST call to generate a token, as follows:

var result = await client.PostAsync(uriBuilder.Uri.AbsoluteUri, null);

When the request has been sent, we expect there to be a response. We want to read this response and return the response string:

return await result.Content.ReadAsStringAsync();

Add a new file called TextToSpeak.cs, if you have not already done so. Put this file in the Model folder.

Beneath the newly created class (but inside the namespace), we want to add two event argument classes. These will be used to handle audio events, which we will see later.

The AudioEventArgs class simply takes a generic stream, as shown in the following code. You can imagine it being used to send the audio stream to our application:

    public class AudioEventArgs : EventArgs
    {
        public AudioEventArgs(Stream eventData)
        {
            EventData = eventData;
        }

        public StreamEventData { get; private set; }
    }

The next class allows us to send an event with a specific error message:

    public class AudioErrorEventArgs : EventArgs
    {
        public AudioErrorEventArgs(string message)
        {
            ErrorMessage = message;
        }

        public string ErrorMessage { get; private set; }
    }

We move on to start on the TextToSpeak class, where we start off by declaring some events and class members, as follows:

    public class TextToSpeak
    {
        public event EventHandler<AudioEventArgs>OnAudioAvailable;
        public event EventHandler<AudioErrorEventArgs>OnError;

        private string _gender;
        private string _voiceName;
        private string _outputFormat;
        private string _authorizationToken;
        private AccessTokenInfo _token;

        private List<KeyValuePair<string, string>> _headers = new  List<KeyValuePair<string, string>>();

The first two lines in the class are events that use the event argument classes that we created earlier. These events will be triggered if a call to the API finishes (returning some audio), or if anything fails. The next few lines are string variables, which we will use as input parameters. We have one line to contain our access token information. The last line creates a new list, which we will use to hold our request headers.

We add two constant strings to our class, as follows:

private const string RequestUri =  "https://speech.platform.bing.com/synthesize";

private const string SsmlTemplate =
    "<speak version='1.0'xml:lang='en-US'>
        <voice xml:lang='en-US'xml:gender='{0}'
        name='{1}'>{2}
        </voice>
    </speak>";

The first string contains the request URI. That is the REST API endpoint that we need to call to execute our request. Next, we have a string defining our Speech Synthesis Markup Language (SSML) template. This is where we will specify what the speech service should say, and how it should say it.

Next, we create our constructor, as follows:

        public TextToSpeak()
        {
            _gender = "Female";
            _outputFormat = "riff-16khz-16bit-mono-pcm";
            _voiceName = "Microsoft Server Speech Text to Speech Voice (en-US, ZiraRUS)";
        }

Here, we are just initializing some of the variables that we declared earlier. As you may see, we are defining the voice as female and we define it so that it uses a specific voice. In terms of gender, it can be either female or male. The voice name can be one of a long list of options. We will look more into the details of that list when we go through this API in a later chapter.

The last line specifies the output format of the audio. This will define the format and codec in use by the resultant audio stream. Again, this can be a number of varieties, which we will look into in a later chapter.

Following the constructor, there are three public methods that we will create. These will generate an authentication token and some HTTP headers, and finally execute our call to the API. Before we create these, you should add two helper methods to be able to raise our events. Call them the RaiseOnAudioAvailable and RaiseOnError methods. They should accept AudioEventArgs and AudioErrorEventArgs as parameters.

Next, add a new method called the GenerateHeaders method, as follows:

        public void GenerateHeaders()
        {
            _headers.Add(new KeyValuePair<string, string>("Content-Type", "application/ssml+xml"));
            _headers.Add(new KeyValuePair<string, string>("X-Microsoft-OutputFormat", _outputFormat));
            _headers.Add(new KeyValuePair<string, string>("Authorization", _authorizationToken));
            _headers.Add(new KeyValuePair<string, string>("X-Search-AppId", Guid.NewGuid().ToString("N")));
            _headers.Add(new KeyValuePair<string, string>("X-Search-ClientID", Guid.NewGuid().ToString("N")));
            _headers.Add(new KeyValuePair<string, string>("User-Agent", "Chapter1"));
        }

Here, we add the HTTP headers to our previously created list. These headers are required for the service to respond, and if any are missing, it will yield an HTTP/400 response. We will cover what we are using as headers in more detail later. For now, just make sure that they are present.

Following this, we want to add a new method called GenerateAuthenticationToken, as follows:

        public bool GenerateAuthenticationToken(string clientSecret)
        {
            Authentication auth = new Authentication(clientSecret);

This method accepts one string parameter, the client secret (your API key). First, we create a new object of the Authentication class, which we looked at earlier, as follows:

        try
        {
            _token = auth.Token;

            if (_token != null)
            {
                _authorizationToken = $"Bearer {_token}";

                return true;
            }
            else
            {
                RaiseOnError(new AudioErrorEventArgs("Failed to generate authentication token."));
                return false;
            }
        }

We use the authentication object to retrieve an access token. This token is used in our authorization token string, which, as we saw earlier, is being passed on in our headers. If the application for some reason fails to generate the access token, we trigger an error event.

Finish this method by adding the associated catch clause. If any exceptions occur, we want to raise a new error event.

The last method that we need to create in this class is going to be called the SpeakAsync method, as shown in the following screenshot. This method will actually perform the request to the Speech API:

        public Task SpeakAsync(string textToSpeak, CancellationTokencancellationToken)
        {
            varcookieContainer = new CookieContainer();
            var handler = new HttpClientHandler() {
                CookieContainer = cookieContainer
            };
            var client = new HttpClient(handler);

The method takes two parameters. One is the string, which will be the text that we want to be spoken. The next is cancellationToken; this can be used to propagate the command that the given operation should be cancelled.

When entering the method, we create three objects that we will use to execute the request. These are classes from the .NET library. We will not be going through them in any more detail.

We generated some headers earlier, and we need to add these to our HTTP client. We do this by adding the headers in the preceding foreach loop, basically looping through the entire list, as shown in the following code:

            foreach(var header in _headers)
            {
                client.DefaultRequestHeaders.TryAddWithoutValidation (header.Key, header.Value);
            }

Next, we create an HTTP Request Message, specifying the request URI and the fact that we will send data through the POST method. We also specify the content using the SSML template that we created earlier, adding the correct parameters (gender, voice name, and the text we want to be spoken), as shown in the following code:

            var request = new HttpRequestMessage(HttpMethod.Post, RequestUri)
            {
                Content = new StringContent(string.Format(SsmlTemplate, _gender, _voiceName, textToSpeak))
            };

We use the HTTP client to send the HTTP request asynchronously, as follows:

            var httpTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken);

The following code is a continuation of the asynchronous send call that we made previously. This will run asynchronously as well, and check the status of the response. If the response is successful, it will read the response message as a stream and trigger the audio event. If everything succeeds, then that stream should contain our text in spoken words:

    var saveTask = httpTask.ContinueWith(async (responseMessage, token) =>
    {
        try
        {
            if (responseMessage.IsCompleted &&
                responseMessage.Result != null &&  
                responseMessage.Result.IsSuccessStatusCode) {
                var httpStream = await responseMessage. Result.Content.ReadAsStreamAsync().ConfigureAwait(false);
                RaiseOnAudioAvailable(new AudioEventArgs (httpStream));
            } else {
                RaiseOnError(new AudioErrorEventArgs($"Service returned {responseMessage.Result.StatusCode}"));
            }
        }
        catch(Exception e)
        {
            RaiseOnError(new AudioErrorEventArgs (e.GetBaseException().Message));
        }
    }

If the response indicates anything other than success, we will raise the error event.

We also want to add a catch clause and a finally clause to this. Raise an error if an exception is caught and dispose of all objects used in the finally clause.

The final code we need specifies that the continuation task is attached to the parent task. We also need to add cancellationToken to this task. Add the following code to finish off the method:

    }, TaskContinuationOptions.AttachedToParent, cancellationToken);
    return saveTask;
}

With this in place, we are now able to utilize this class in our application. Open the MainViewModel.cs file and declare a new class variable, as follows:

        private TextToSpeak _textToSpeak;

Add the following code in the constructor to initialize the newly added object. We also need to call a function to generate the authentication token, as follows:

            _textToSpeak = new TextToSpeak();
            _textToSpeak.OnAudioAvailable +=  _textToSpeak_OnAudioAvailable;
            _textToSpeak.OnError += _textToSpeak_OnError;

            GenerateToken();

After we have created the object, we hook up the two events to event handlers. Then we generate an authentication token by creating a GenerateToken function with the following content:

public async void GenerateToken()
{
    if (await _textToSpeak.GenerateAuthenticationToken("BING_SPEECH_API_KEY_HERE"))
        _textToSpeak.GenerateHeaders();
}

Then we generate an authentication token, specifying the API key for the Bing Speech API. If that call succeeds, we generate the HTTP headers required.

We need to add the event handlers, so create the _textToSpeak_OnError method first, as follows:

            private void _textToSpeak_OnError(object sender, AudioErrorEventArgs e)
            {
                StatusText = $"Status: Audio service failed -  {e.ErrorMessage}";
            }

It should be a rather simple method, just outputting the error message to the user in the status text field.

Next, we need to create a _textToSpeak_OnAudioAvailable method, as follows:

        private void _textToSpeak_OnAudioAvailable(object sender, AudioEventArgs e)
        {
            SoundPlayer player = new SoundPlayer(e.EventData);
            player.Play();
            e.EventData.Dispose();
        }

Here, we utilize the SoundPlayer class from the .NET framework. This allows us to add the stream data directly and simply play the message.

The last part that we need for everything to work is to make the call to the SpeakAsync method. We can make this by adding the following at the end of our DetectFace method:

    await _textToSpeak.SpeakAsync(textToSpeak, CancellationToken.None);

With that in place, you should now be able to compile and run the application. By loading a photo and clicking on Detect face, you should be able to get the number of faces in the image spoken back to you. Just remember to have your audio turned on!

This chapter was a brief introduction to Microsoft Cognitive Services. We started off by creating a template project to easily create new projects for the coming chapters. We tried this template by creating an example project for this chapter. Then you learned how to detect faces in images by utilizing the Face API. From there, we took a quick tour of what Cognitive Services has to offer. We finished off by adding text-to-speech capabilities to our application by using the Bing Speech API.

The next chapter will go into more detail of the vision part of the APIs. There, you will learn how to analyze images using the computer vision API. You will go into more detail about the Face API and will learn how to detect emotions in faces by using the emotion API. We will use some of this to start building our smart-house application.