Discovering the feature store
The feature store is a relatively new yet stable release in the latest Databricks ML workspace. Many organizations that have mature ML processes in place, such as Uber, Facebook, DoorDash, and many more, have internally implemented their feature stores.
ML life cycle management and workflows are complex. Forbes conducted a survey (https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says) with data scientists and uncovered that managing data is the most expensive and time-consuming operation in their day-to-day work.
Data scientists need to spend a lot of time finding the data, cleaning it, doing EDA, and then performing feature engineering to train their ML models. This is an iterative process. The effort that needs to be put in to make the process repeatable is an enormous challenge. This is where feature stores come in.
