Chapter 11: Implementing a Real-Time Scoring Solution

While most machine learning (ML) projects involve batch scoring, the most complex ML projects use real-time solutions. Think about models that determine whether a credit card transaction is fraudulent, models that decide which ads to show online shoppers, and models that decide whether a customer at a car dealership is creditworthy or not. These situations all demand a real-time scoring solution and it's incredibly important that your model be both fast and accurate.

Luckily, creating a fast, reliable real-time scoring solution in AutoML is easy whether you decide to code it with Python or use the Azure Machine Learning (AML) Studio graphical user interface (GUI).

You will begin this chapter by creating a real-time scoring endpoint through the AML studio GUI. Real-time scoring endpoints are web services through which you can pass data and quickly receive results. Continuing, you will then create real-time scoring endpoints...

Technical requirements

In this chapter, you will be creating an Azure Kubernetes Service (AKS) instance through AML studio and creating real-time scoring endpoints using the Diabetes-AllData-Regression-AutoML that you created in Chapter 4, Building an AutoML Regression Solution.

As such, you will need a working internet connection, an Azure Machine Learning Service (AMLS) workspace, and a compute instance. You will also need permission to create an AKS cluster. If you are using a personal account, this will not be an issue.

The following are the prerequisites for the chapter:

• Have access to the internet
• Have a web browser, preferably Google Chrome or Microsoft Edge Chromium
• Have a Microsoft Azure account
• Have created an AMLS workspace
• Have created the compute-cluster compute cluster in Chapter 2, Getting Started with Azure Machine Learning Service
• Understand how to navigate to the Jupyter environment from an Azure compute instance as demonstrated...

Creating real-time endpoints through the UI

The crux of any real-time scoring solution is a real-time scoring endpoint, a web URL through which you can pass data and immediately retrieve ML predictions. Endpoints are hosted on containerized services that are up and running 24 hours a day, 7 days a week, waiting for incoming requests.

Requests send data to the endpoint for scoring and can be written in any computer language including Python. As soon as a request comes through, your endpoint will automatically execute the underlying code and return results.

You can use these endpoints anywhere; any coding language from C# to Python to Java can make use of real-time scoring endpoints. Thus, once you obtain the URL that hosts the endpoint, you are free to implement it in any other piece of code. Commonly, real-time scoring endpoints are incorporated in streaming jobs, web applications, and mobile apps.

When using real-time scoring endpoints based on AutoML models, there are...

Creating real-time endpoints through the SDK

One-click deployment through AML studio is really easy, but most organizations will require you to develop your solutions via code. Luckily, creating real-time scoring endpoints for AutoML models via the AzureML Python SDK is almost as easy as creating them through the UI. Furthermore, you'll gain a deeper understanding of how your endpoints work and how to format your JSON testing to pass data into the endpoint as a request.

In this section, you'll begin by entering your Jupyter environment and creating a new notebook. First, you will deploy your Diabetes-AllData-Regression-AutoML model via ACI, test it, and, once you've confirmed that your test is a success, create a new AKS cluster via code and deploy it there. You will conclude this section by testing your AKS deployment and confirm that everything works as expected.

The goal of this section is to further your understanding of real-time scoring endpoints, teach...

Improving performance on your AKS cluster

Sometimes you will deploy an endpoint on AKS and it doesn't perform how you'd like. Maybe it times out, maybe it's too slow, maybe an endpoint that was previously working fine suddenly gets a lot more traffic that it cannot handle. These situations happen, and you must be prepared to face them.

Thankfully, AKS deployments have a lot of additional configurations that you can take advantage of to solve these problems. This section covers some of the more common situations as follows:

• Depending on how complex your model is, how many data points you are trying to score, and the size of your VMs, AKS models can sometimes take a while to score or even timeout. In this situation, there are many things you can do.

  First, you can try increasing the size of your VM, selecting one with more RAM. Next, you can add an additional setting to your deployment configuration, scoring_timeout_ms. This setting defaults to 60000 milliseconds...

Summary

You have now created and tested real-time scoring solutions using an AutoML trained model. Deploying first on ACI and then on AKS, you understand the full end-to-end process of creating a real-time scoring endpoint.

Furthermore, you understand how data must be shaped and formatted in order to generate predictions using these endpoints, which can be incorporated into any piece of code using a wide variety of computer languages to create powerful, innovative solutions.

In the next chapter, Chapter 12, Realizing Business Value with AutoML, the final chapter of the book, you will learn how to present AutoML solutions in a way that will gain the trust of your non-technical business partners. Their trust and acceptance, after all, is the foundation to unlocking the power and value of ML and artificial intelligence in your organization.