The p-value and its role in conformal prediction

In conformal prediction, p-values are key in constructing prediction regions and intervals with a guaranteed confidence level. However, their purpose is different than in traditional statistical hypothesis testing.

Let’s walk through an example binary classification task to understand how this works. Suppose we want to predict whether a patient has a medical condition based on their symptoms and characteristics:

1. First, we calculate a nonconformity score that measures how different or “nonconforming” the new patient is compared to previously seen patients. We can define this score in various ways, such as the distance between feature values.
2. Next, we temporarily assign the patient each possible label – 0 (no condition) and 1 (has condition) – and recalculate the nonconformity score with that assigned label.

If the score is similar to scores for past patients with label 0, then label...