Types of Conformal Predictors

This chapter describes different families of conformal predictors, exploring various approaches to quantifying uncertainty. Through practical examples, we provide an intermediate-level understanding of these techniques and how they can be applied to real-world situations.

Here are examples of how companies are using conformal prediction.

At a high-profile AI developer conference called GTC 2023 (https://www.nvidia.com/gtc/), Bill Dally, NVIDIA’s chief scientist and SVP of research, offered insights into one of NVIDIA’s R&D primary focuses, which is in conformal prediction (https://www.hpcwire.com/2023/03/28/whats-stirring-in-nvidias-rd-lab-chief-scientist-bill-dally-provides-a-peek/).

Traditional machine learning models for autonomous vehicles output a single classification (e.g., pedestrian or no pedestrian on the road) and position estimate for detected objects. However, NVIDIA wants to produce a set of potential outputs with...

Understanding classical predictors

Before we deep dive into the intricacies of conformal predictors, let’s briefly recap the key concepts from the previous chapters. Conformal prediction is a framework that enables creating confidence regions for our predictions while controlling the error rate.

This approach is especially beneficial in situations where a measure of uncertainty is essential, such as in medical diagnosis, self-driving cars, or financial risk management. The framework encompasses two main types of conformal predictors: classical and inductive.

Classical transductive conformal prediction (TCP) is the original form of conformal prediction developed by the inventors of Conformal prediction. It forms the basis for understanding the general principles of conformal predictors. Classical Conformal prediction was developed to construct prediction regions that conform to a specified confidence level. The critical aspect of classical Conformal prediction is its distribution...

Understanding inductive conformal predictors

ICP is a variant of conformal prediction that provides valid predictive regions under the same assumptions as classical conformal prediction and has the added benefit of improved computational efficiency, which is particularly useful when dealing with large datasets.

ICPs present a highly efficient and effective solution within the realm of machine learning. They provide a form of conformal prediction that caters to larger datasets, making it highly suitable for real-world applications that involve extensive data volumes. ICPs divide the dataset into training and calibration sets during the model-building process. The training set is used to develop the model, while the calibration set helps calculate the nonconformity scores. This two-step process optimizes computation and delivers precise prediction regions.

Figure 5.3 – Inductive conformal prediction

A predictive model, such as a neural network...

Choosing the right conformal predictor

Both classical and inductive conformal predictors offer valuable approaches to building reliable machine learning models. However, they each come with unique strengths and weaknesses.

Classical transductive conformal predictors are highly adaptable and do not make any assumptions about data distribution. However, they tend to be computationally expensive, requiring the model’s retraining for each new prediction.

Inductive conformal predictors, conversely, are computationally more efficient, as they only require the model to be trained once.

Choosing the right conformal predictor largely depends on the specific requirements of the problem at hand. Some considerations might include the following:

• Computation resources: If computation resources or time are a concern, inductive conformal predictors might be more suitable due to their reduced computational cost
• Data size: For smaller datasets, classical conformal predictors...

Summary

This chapter explored the fascinating world of conformal predictors, their types, and their distinctive features. The key concepts and skills we touched upon include covering the foundational principles of conformal prediction and its application in machine learning. It also highlighted the differences between classical transductive and inductive conformal predictors. We also covered how to effectively choose the appropriate type of conformal predictor based on the specific requirements of the problem. Finally, the practical applications of conformal predictors in binary classification, multiclass classification, and regression were also included.

The chapter also provided a detailed algorithmic description and mathematical formulation of classical and inductive conformal predictors, adding to our theoretical understanding. To deepen our learning, we also took a hands-on approach, looking at practical examples in Python.

For those interested in further exploring conformal...