Scaling Up

The previous chapter was all about starting the conversation around how we get our solutions out into the world using different deployment patterns, as well as some of the tools we can use to do this. This chapter will aim to build on that conversation by discussing the concepts and tools we can use to scale up our solutions to cope with large volumes of data or traffic.

Running some simple machine learning (ML) models on a few thousand data points on your laptop is a good exercise, especially when you’re performing the discovery and proof-of-concept steps we outlined previously at the beginning of any ML development project. This approach, however, is not appropriate if we have to run millions upon millions of data points at a relatively high frequency, or if we have to train thousands of models of a similar scale at any one time. This requires a different approach, mindset, and toolkit.

In the following pages, we will cover some details of two of the most...

Join our book community on Discord

https://packt.link/EarlyAccessCommunity

In this chapter, we will dive into some important concepts around the deployment of your Machine Learning (ML) solution. We will begin to close the circle of the ML development life cycle and lay the groundwork for getting your solutions out into the world.

The act of deploying software, of taking it from a demo you can show off to a few stakeholders to a service that will ultimately impact customers or colleagues, is a very exhilarating but often challenging exercise. It also remains one of the most difficult aspects of any ML project and getting it right can ultimately make the difference between generating value or just hype.

We are going to explore some of the main concepts that will help your ML engineering team cross the chasm between a fun proof-of-concept to solutions that can run on scalable infrastructure in an automated way.

In this chapter, we will cover the following topics:

• Architecting systems...

Technical requirements

To work through the examples in this chapter, we require the following tools to be installed:

• AWS CLI v2
• Postman
• Docker

Architecting systems

No matter how you are working to build your software, it is always important to have a design in mind. This section will highlight the key considerations we must bear in mind when architecting ML systems.

Consider a scenario where you are contracted to organize the building of a house. We would not simply go out and hire a team of builders, buy all the supplies, hire all the equipment, and just tell everyone to start building. We would also not assume we knew exactly what the client who hired us wants without first speaking to them.

Instead, we would likely try to understand what the client wanted in detail, and then try to design the solution that would fit their requirements. We would potentially iterate this plan a few times with them and with appropriate experts who knew the details of pieces that fed into the overall design. Although we are not interested in building houses (or maybe you are, but there will not be any in this book!), we can still see the analogy...

Exploring the unreasonable effectiveness of patterns

In this book, we have already mentioned a few times that we should not attempt to reinvent the wheel and we should reuse, repeat, and recycle what works according to the wider software and ML community. This is also true about your deployment architectures. When we discuss architectures that can be reused for a variety of different use cases with similar characteristics, we often refer to these as patterns. Using standard (or at least well-known) patterns can really help you speed up the time to value of your project and help you engineer your ML solution in a way that is robust and extensible.

Given this, we will spend the next few sections summarizing some of the most important architectural patterns that have become increasingly successful in the ML space over the past few years.

Swimming in data lakes

The single most important asset for anyone trying to use ML is, of course, the data that we can analyze and train our models on...

Containerizing

If you develop software that you want to deploy somewhere, which is the core aim of an ML engineer, then you have to be very aware of the environmental requirements of your code, and how different environments might affect the ability of your solution to run. This is particularly important for Python, which does not have a core capability for exporting programs as standalone executables (although there are options for doing this). This means that Python code needs a Python interpreter to run and needs to exist in a general Python environment where the relevant libraries and supporting packages have been installed.

A great way to avoid headaches from this point of view is to ask the question: Why can't I just put everything I need into something that is relatively isolated from the host environment, which I can ship and then run as a standalone application or program? The answer to this question is that you can and that you do this through containerization. This is...

Hosting your own microservice on AWS

A classic way to surface your ML models is via a lightweight web service hosted on a server. This can be a very flexible pattern of deployment. You can run a web service on any server with access to the internet (roughly) and, if designed well, it is often easy to add further functionality to your web service and expose it via new endpoints.

In Python, the two most used web frameworks have always been Django and Flask. In this section, we will focus on Flask as it is the simpler of the two and has been written about extensively for ML deployments on the web, so you will be able to find plenty of material to build on what you learn here.

On AWS, one of the simplest ways you can host your Flask web solution is as a containerized application on an appropriate platform. We will go through the basics of doing this here, but we will not spend time on the detailed aspects of maintaining good web security for your service. To fully discuss this may require...

5 Deployment Patterns and Tools

Join our book community on Discord

https://packt.link/EarlyAccessCommunity

In this chapter, we will dive into some important concepts around the deployment of your Machine Learning (ML) solution. We will begin to close the circle of the ML development life cycle and lay the groundwork for getting your solutions out into the world.

The act of deploying software, of taking it from a demo you can show off to a few stakeholders to a service that will ultimately impact customers or colleagues, is a very exhilarating but often challenging exercise. It also remains one of the most difficult aspects of any ML project and getting it right can ultimately make the difference between generating value or just hype.

We are going to explore some of the main concepts that will help your ML engineering team cross the chasm between a fun proof-of-concept to solutions that can run on scalable infrastructure in an automated way.

In this chapter, we will cover the following...