Elements of a Machine Learning System

Data and algorithms are crucial for machine learning systems, but they are far from sufficient. Algorithms are the smallest part of a production machine learning system. Machine learning systems also require data, infrastructure, monitoring, and storage to function efficiently. For a large-scale machine learning system, we need to ensure that we can include a good user interface or package model in microservices.

In modern software systems, combining all necessary elements requires different professional competencies – including machine learning/data science engineering expertise, database engineering, software engineering, and finally interaction design. In these professional systems, it is more important to provide reliable results that bring value to users rather than include a lot of unnecessary functionality. It is also important to orchestrate all elements of machine learning together (data, algorithms, storage, configuration,...

Elements of a production machine learning system

Modern machine learning algorithms are very capable because they use large quantities of data and consist of a large number of trainable parameters. The largest available models are Generative Pre-trained Transformer-3 (GPT-3) from OpenAI (with 175 billion parameters) and Megatron-Turing from NVidia (356 billion parameters). These models can create texts (novels) and make conversations but also write program code, create user interfaces, or write requirements.

Now, such large models cannot be used on a desktop computer, laptop, or even in a dedicated server. They need advanced computing infrastructure, which can withstand long-term training and evaluation of such large models. Such infrastructure also needs to provide means to automatically provide these models with data, monitor the training process, and, finally, provide the possibility for the users to access the models to make inferences. One of the modern ways of providing such...

Data and algorithms

Now, if using the algorithms is not the main part of the machine learning code, then something else must be – that is, data handling. Managing data in machine learning software, as shown in Figure 2.1, consists of three areas:

1. Data collection.
2. Feature extraction.
3. Data validation.

Although we will go back to these areas throughout this book, let’s explore what they contain. Figure 2.2 shows the processing pipeline for these areas:

Figure 2.2 – Data collection and preparation pipeline

Note that the process of preparing the data for the algorithms can become quite complex. First, we need to extract data from its source, which is usually a database. It can be a database of measurements, images, texts, or any other raw data. Once we’ve exported/extracted the data we need, we must store it in a raw data format. This can be in the form of a table, as shown in the preceding figure, or it can...

Data collection

Data collection is a procedure of transforming data from its raw format to a format that a machine learning algorithm can take as input. Depending on the data and the algorithm, this process can take different forms, as illustrated in Figure 2.3:

Figure 2.3 – Different forms of data collection – examples

Data from images and measurements such as time series is usually collected to make classifications and predictions. These two classes of problems require the ground truth to be available, which we saw as Y_train in the previous code example. These target labels are either extracted automatically from the raw data or added manually through the process of labeling. The manual process is time-consuming, so the automated one is preferred.

The data that’s used in non-supervised learning and reinforcement learning models is often extracted as tabular data without labels. This data is used in the decision process or the...

Configuration and monitoring

Machine learning software is meant to be professionally engineered, deployed, and maintained. Modern companies call this process MLOps, which means that the same team needs to take responsibility for both the development and operations of the machine learning system. The rationale behind this extended responsibility is that the team knows the system best and therefore can configure, monitor, and maintain it in the best possible way. The teams know the design decisions that must be taken when developing the system, assumptions made about the data, and potential risks to monitor after the deployment.

Configuration

Configuration is one such design decision that’s made by the development team. The team configures the parameters of the machine learning models, the execution environment, and the monitoring infrastructure. Let’s explore the first one; the latter two will be discussed in the next few sections.

To exemplify this challenge...

Infrastructure and resource management

The infrastructure and resources needed for the machine learning software are organized into two areas – data serving infrastructure (for example, databases) and computational infrastructure (for example, GPU computing platforms). There is also serving infrastructure, which is used to provide the services to the end users. The serving infrastructure could be in the form of desktop applications, embedded software (such as the one in autonomous vehicles), add-ins to tools (as in the case of GitHub Co-pilot), or websites (such as ChatGPT). However, in this book, we’ll focus on the data-serving infrastructure and the computational infrastructure.

Both areas can be deployed locally or remotely. Local deployment means that we use our own infrastructure at the company, while remote infrastructure means that we use cloud services or services of another supplier.

Conceptually, we could see these two areas as co-dependent, as depicted...

How this all comes together – machine learning pipelines

In this chapter, we explored the main characteristics of machine learning systems and compared them to traditional software systems. Let’s finish this comparison by summarizing how we usually design and describe machine learning systems – by using pipelines. A pipeline is a sequence of data processing steps, including the machine learning models. The typical set of steps (also called phases) is shown in Figure 2.14:

Figure 2.14 – A typical sequence of steps in a machine learning pipeline

This kind of pipeline, although drawn linearly, is usually processed in cycles, where, for example, monitoring for concept drift can trigger re-training, re-testing, and re-deployment.

Machine learning pipelines, just like the one presented in Figure 2.14, are often depicted as a set of components as parts of the entire system. However, presenting it using the pipeline analogy helps...

References

• Shortliffe, E.H., et al., Computer-based consultations in clinical therapeutics: explanation and rule acquisition capabilities of the MYCIN system. Computers and biomedical research, 1975. 8(4): p. 303-320.
• Vaswani, A., et al., Attention is all you need. Advances in neural information processing systems, 2017. 30.
• Dale, R., GPT-3: What’s it good for? Natural Language Engineering, 2021. 27(1): p. 113-118.
• Smith, S., et al., Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022.
• Lee, Y.W., et al., AIMQ: a methodology for information quality assessment. Information & management, 2002. 40(2): p. 133-146.
• Zenisek, J., F. Holzinger, and M. Affenzeller, Machine learning based concept drift detection for predictive maintenance. Computers & Industrial Engineering, 2019. 137: p. 106031.
• Amershi, S., et al. Software engineering for machine learning: A...