The influence of data balancing techniques on model calibration

The usual impact of applying data-level techniques, such as oversampling and undersampling, is that they change the distribution of the training data for the model. This means that the model sees an almost equal number of all the classes, which doesn’t reflect the actual data distribution. Because of this, the model becomes less calibrated against the true imbalanced distribution of data. Similarly, algorithm-level cost-sensitive techniques that use class_weight to account for the data imbalance have a similar degraded impact on degrading the calibration of the model against the true data distribution. Figure 10.7 (log scale) from a recent study [7] shows the degrading calibration of a CNN-based model for pneumonia detection task, as class_weight increases from 0.5 to 0.9 to 0.99. The model becomes over-confident and hence less calibrated with the increase in class_weight.

Figure 10.7...