Overview of performance monitoring for machine learning models
Monitoring is at the cornerstone of reliable ML systems able to consistently unlock the value of data and provide critical feedback for improvement.
On the monitoring side of ML models, there are multiple interested parties, and we should take the requirements for monitoring from the different stakeholders involved. One example of a typical set of stakeholders is the following:
- Data scientists: Their focus regarding monitoring is evaluating model performance and data drift that might negatively affect that performance.
- Software engineers: These stakeholders want to ensure that they have metrics that assess whether their products have reliable and correct access to the APIs that are serving models.
- Data engineers: They want to ensure that the data pipelines are reliable and pushing data reliably, at the right velocity, and in line with the correct schemas.
- Business/product stakeholders: These stakeholders...
