Where different types of data can be used together – an outlook on multi-modal data models

This chapter introduced three types of data – images, text, and structured text. These three types of data are examples of data that is in a numerical form, such as matrices of numbers, or in forms of time series. Regardless of the form, however, working with data and ML systems is very similar. We need to extract the data from a source system, then transform it into a format that we can annotate, and then use this as input to an ML model.

When we consider different types of data, we could start to think about whether we could use two types of data in the same system. There are a few ways of doing that. The first one is when we use different ML systems in different pipelines, but we connect the pipelines. GitHub Copilot is such a system. It uses a pipeline for processing a natural language to find similar programs and to transform them so that they fit the context of the program...