8.3 Being robust against dataset shift

We already encountered dataset shift in Chapter 3, Fundamentals of Deep Learning. As a reminder, dataset shift is a common problem in machine learning that happens when the joint distribution P(X,Y ) of inputs X and outputs Y differs between the model training stage and model inference stage (for example, when testing the model or when running it in a production environment). Covariate shift is a specific case of dataset shift where only the distribution of the inputs changes but the conditional distribution P(Y |X) stays constant.

Dataset shift is present in most production environments because of the difficulty of including all possible inference conditions during training and because most data is not static but changes over time. The input data can shift along many different dimensions in a production environment. Geographic and temporal dataset shift are two common forms of shift. Imagine, for example, you have trained your model on data...