Deploying Models to Production

In the previous chapters of this book, we've worked on our skills for developing, testing, and using various deep learning models. We haven't talked much about the role of deep learning within the broader context of software engineering. In this last chapter, we will use the time to talk about continuous delivery, and the role of machine learning within this context. We will then look at how you can deploy models to production with a continuous delivery mindset. Finally, we will look at Azure Machine Learning service to properly manage the models you develop.

The following topics will be covered in this chapter:

• Using machine learning in a DevOps environment
• Storing models
• Using Azure Machine Learning service to manage models

Technical requirements

We assume that you have a recent version of Anaconda installed on your computer, and have followed the steps in Chapter 1, Getting Started with CNTK, to install CNTK on your computer. The sample code for this chapter can be found in our GitHub repository at: https://github.com/PacktPublishing/Deep-Learning-with-Microsoft-Cognitive-Toolkit-Quick-Start-Guide/tree/master/ch7.

In this chapter, we'll work on a few examples stored in Jupyter notebooks. To access the sample code, run the following commands inside an Anaconda prompt in the directory where you've downloaded the code:

cd ch7
jupyter notebook

This chapter also contains a C# code sample to demonstrate how to load models in the open source ONNX format. If you want to run the C# code you will need to have .NET Core 2.2 installed on your machine. You can download the latest version of .NET core...

Using machine learning in a DevOps environment

Most modern software development happens in an agile fashion, in an environment where developers and IT-pros work on the same project. The software we're building often is deployed to production through continuous integration and continuous deployment pipelines. How are we going to integrate machine learning in this modern environment? And does it mean we have to change a lot when we start building AI solutions? These are some of the frequently asked questions you can run into when you introduce AI and machine learning to the workflow.

Luckily, you don't have to change your whole build environment or deployment tool stack to integrate machine learning into your software. Most of the things that we'll talk about will fit right into your existing environment.

Let's take a look at a typical continuous delivery scenario...

Storing your models

In order to be able to deploy your models to production, you need to be able to store a trained model on disk. CNTK offers two ways to store models on disk. You can either store checkpoints to continue training at a later time, or you can store a portable version of your model. Each of these storage methods has its own use.

Storing model checkpoints to continue training at a later point

Some models take a long time to train, sometimes up to weeks at a time. You don't want to lose all your progress when your machine crashes during training, or if there's a power outage.

This is where checkpointing becomes useful. You can create a checkpoint during training using a CheckpointConfig object. You...

Using Azure Machine Learning service to manage models

While you can completely hand-build a continuous integration pipeline, it's still quite a bit of work. You need to get dedicated hardware to run deep learning training jobs, and that can bring up the costs. There are great alternatives available in the cloud. Google has a TensorFlow serving offer. Microsoft offers Azure Machine Learning service as a way to manage models. Both are great tools that we can highly recommend.

Let's take a look at Azure Machine Learning service to get a sense of what it can do for you when you want to set up a complete machine learning pipeline:

Azure Machine Learning service is a cloud service that offers a complete solution for every phase of your machine learning project. It has the concept of experiments, and runs that allow you to manage experiments. It features a model registry that...

Summary

In this chapter, we've looked at what it takes to bring deep learning and machine learning models to production. We've explored some of the basic principles that will help you to be successful with deep learning in a continuous delivery environment.

We've taken a look at exporting models to ONNX to make it easier to deploy your trained models to production and keep them running for years, thanks to the portable nature of the ONNX format. We then explored how you can use the CNTK API in other languages, such as C#, to make predictions.

Finally, we've looked at using Azure Machine Learning service to level-up your DevOps experience with experiment management, model management, and deployment tools. Although you don't need a tool like this to get started, it really helps to have something like Azure Machine Learning service in your arsenal when...