Managing Drift Effectively in a Dynamic Environment

Drift is a significant factor in the performance deterioration of deployed deep learning models over time, encompassing concept drift, data drift, and model drift. Let’s understand the drift of a deployed model through a culinary-based analogy. Imagine a deployed deep learning model as a skilled chef who aims to create dishes that delight customers but excels in a particular cuisine. Concept drift occurs when the taste preferences of the diner shift, which alters the relationships between ingredients and popular dishes that can satisfy the diner’s palate. Data drift, on the other hand, happens when the ingredients themselves change, such as variations in flavor or availability. Finally, model metric monitoring alerts happen most straightforwardly when the chef loses customers. In all cases, the chef must adapt their dishes to maintain their success, just as deep learning models need to be updated to account for concept...

Technical requirements

This chapter will cover a practical example to test out data drift techniques. We will be using Python 3.10 and, additionally, we will require the following Python libraries to be installed:

• evidently
• numpy
• transformers==4.21.3
• torch==1.12.0
• syllables
• audiomentations
• datasets

The code files are available on GitHub: https://github.com/PacktPublishing/The-Deep-Learning-Architect-Handbook/tree/main/CHAPTER_17.

Exploring the issues of drift

The most obvious issue of drift is the degradation of the accuracy. However, there are more issues than you might initially notice, which include the following:

• Applicability: The model’s ability to make accurate predictions on new, unseen data may be compromised as data patterns and distributions shift. This can result in reduced effectiveness in real-world scenarios and diminished value for decision-making, which raises the likelihood of the model becoming less relevant and practical to use.
• Interpretability: Understanding and explaining the model’s decisions can become challenging, as the factors influencing its predictions may no longer align with the current data landscape. This can hinder effective communication with stakeholders and impede trust in the model’s predictions. Note that an originally explainable model is still explainable as we can still produce accurate information on how it used the input data, but...

Exploring the types of drift

Drift is like a shift in the way things work with data. It happens when the data changes, or the environment it comes from changes. This can sometimes happen suddenly or quickly, sometimes slowly, or even in a recurring pattern. When it comes to drift, it’s important to look at the big picture, not just a couple of odd blips. Drift isn’t about those rare anomalies or one or two odd predictions; it’s about changes that stick around, like a new pattern that stays. These persistent shifts can mess up your model permanently, making it way less useful. It’s like if your friend suddenly started speaking a different language occasionally, which could lead to one-off confusion but not really be a problem. But if they started speaking a different language all the time, it’d be a big problem.

Furthermore, drift can be categorized into three main types: data drift, concept drift, and model drift. While concept drift is related...

Exploring strategies to handle drift

Simply setting up drift monitoring for a deployed model isn’t enough to effectively tackle all potential drift-related challenges. It’s crucial to ask yourself: does the specific drift with the chosen data type impact the model’s performance in the metrics that matter the most? At what point does drift become intolerable? To properly address drift, start by pinpointing the drift metric and data type that carries the most significance for your model and the business. If your model has been developed correctly, it may possess generalizable properties, which is the primary goal for most machine learning practitioners. This means that a well-developed model should be able to handle drift effectively. When drift detection and alerts are configured without proper consideration of their effects, it poses the risk that drift alerts can be raised without an actual issue, which can result in wasted time and resources that could have been...

Detecting drift programmatically

With a comprehensive understanding of drift types and their effects, we will explore techniques for detecting drift programmatically, diving into the realms of concept drift and data drift. Armed with these methods, you’ll be well equipped to implement high-risk drift detection components. Let’s start with concept drift.

Detecting concept drift programmatically

Concept drift involves both the input data and the target data. This means that we can effectively detect concept drift for a deployed model only when we can get access to the real target labels in production. When you do have access to them, you can adopt the following techniques to detect concept drift:

• Check the similarity of production data to the reference training data: This should include both input and output data.
• Use model evaluation metrics as a proxy: Evaluation metrics can signal concept drift or data drift.
• Use multivariate-based data drift detection...

Summary

In this chapter, we explored the concept of drift, which affects the performance of deployed deep learning models over time. We covered the three types of drift – concept drift, data drift, and model drift – and discussed strategies to handle them effectively. This included strategies to approach drift, including automatic programmatic detection and manual domain expert predictions, strategies to quantify drift, and strategies to mitigate drift effectively. We learned that statistical-based drift should always be opted for over ambiguous data distribution drift. We also learned that monitoring drift by batch in regular intervals is crucial in ensuring the continued success of deep learning models. Finally, using the evidently library, we demonstrated how to implement programmatic data distribution drift detection in a practical tutorial and understood behaviors that can shape how you think of data distribution drift methods. This knowledge can be applied across...