Introducing Conformal Prediction

This book is about conformal prediction, a modern framework for uncertainty quantification that is becoming increasingly popular in industry and academia.

Machine learning and AI applications are everywhere. In the realm of machine learning, prediction is a fundamental task. Given a training dataset, we train a machine learning model to make predictions on new data.

Figure 1.1 – Machine learning prediction model

However, in many real-world applications, the predictions made by statistical, machine learning, and deep learning models are often incorrect or unreliable because of various factors, such as insufficient or incomplete data, issues arising during the modeling process, or simply because of the randomness and complexities of the underlying problem.

Predictions made by machine learning models often come without the uncertainty quantification required for confident and reliable decision-making. This is...

Technical requirements

This book uses Python. The code for this book is hosted on GitHub and can be found here: https://github.com/PacktPublishing/Practical-Guide-to-Applied-Conformal-Prediction You can run notebooks locally or upload them to Google Colab (https://colab.research.google.com/).

Introduction to conformal prediction

In this section, we will introduce conformal prediction and explain how it can be used to improve the reliability of predictions produced by statistical, machine learning, and deep learning models. We will provide an overview of the key ideas and concepts behind conformal prediction, including its underlying principles and benefits. By the end of this section, you will have a solid understanding of conformal prediction and why it is an important framework to know.

Conformal prediction is a powerful machine learning framework that provides valid confidence measures for individual predictions. This means that when you make a prediction using any model from the conformal prediction framework, you can also measure your confidence in that prediction.

This is incredibly useful in many practical applications where it is crucial to have reliable and interpretable predictions. For example, in medical diagnosis, conformal prediction can provide a confidence...

The origins of conformal prediction

The origins of conformal prediction are documented in Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification by Anastasios N. Angelopoulos and Stephen Bates (https://arxiv.org/abs/2107.07511).

Note

Conformal prediction was invented by my PhD supervisor Prof. Vladimir Vovk, a professor at Royal Holloway University of London. Vladimir Vovk graduated from Moscow State University, where he studied mathematics and became a student of one of the most notable mathematicians of the 20th century, Andrey Kolmogorov. During this time, initial ideas that later gave rise to the invention of conformal prediction appeared.

The first edition of Algorithmic Learning in a Random World (https://link.springer.com/book/10.1007/b106715) by Vladimir Vovk, Alexander Gammerman, and Glenn Shafer was published in 2005. The second edition of the book was published in 2022 (https://link.springer.com/book/10.1007/978-3-031-06649...

How conformal prediction differs from traditional machine learning

Conformal prediction allows the production of well-calibrated probabilistic predictions for any statistical, machine learning, or deep learning model. This is achieved without relying on restrictive assumptions required by other methods such as Bayesian techniques, Monte Carlo simulation, and bootstrapping. Importantly, conformal prediction does not require subjective priors. It provides mathematically guaranteed, well-calibrated predictions every time – regardless of the underlying prediction model, data distribution, or dataset size.

A key limitation of traditional machine learning is the need for more reasonable confidence measures for individual predictions. Models may have excellent overall performance but not be able to quantify uncertainty for a given input reliably.

Conformal prediction solves this by outputting prediction regions and confidence measures with statistical validity guarantees. It...

The p-value and its role in conformal prediction

In conformal prediction, p-values are key in constructing prediction regions and intervals with a guaranteed confidence level. However, their purpose is different than in traditional statistical hypothesis testing.

Let’s walk through an example binary classification task to understand how this works. Suppose we want to predict whether a patient has a medical condition based on their symptoms and characteristics:

1. First, we calculate a nonconformity score that measures how different or “nonconforming” the new patient is compared to previously seen patients. We can define this score in various ways, such as the distance between feature values.
2. Next, we temporarily assign the patient each possible label – 0 (no condition) and 1 (has condition) – and recalculate the nonconformity score with that assigned label.

If the score is similar to scores for past patients with label 0, then label...

Summary

In this chapter, we have introduced conformal prediction and explained the multiple benefits of this powerful framework for reliably quantifying the uncertainty of predictions to improve trust in machine learning models.

We explained that the key benefit of conformal prediction is that, unlike any other probabilistic prediction framework, it provides valid probabilistic predictions accompanied by confidence measures, regardless of the underlying model, the dataset size, and the data distribution.

We then explored the origins of conformal prediction and saw how it has recently become a very popular framework adopted by leading universities and companies.

Finally, we looked at how conformal prediction differs from traditional machine learning and learned about the role of p-values in conformal prediction.

In Chapter 2, we will explain why conformal prediction is a valuable tool for quantifying the uncertainty of predictions, especially in critical settings such as...