It’s All About Data – Options to Store and Transform ML Datasets

The real work on a machine learning project only starts once the required data is available in the project development environment. Sometimes, when the data changes very frequently or the use case requires real-time data, we may need to set up some data pipelines to ensure that the required data is always available for analysis and modeling purposes. The best way to transfer, store, or transform data also depends on the size, type, and nature of the underlying data. Raw data, as collected in the real world, is often massive in size and may belong to multiple types, such as text, audio, images, videos, and so on. Due to the varying nature, size, and type of real-world data, it becomes really important to set up the correct infrastructure for storing, transferring, transforming, and analyzing the data at scale.

In this chapter, we will learn about the different options for moving data to the Google Cloud...

Moving data to Google Cloud

When we start a machine learning project on Google Cloud Platform (GCP), the very first step is to move all our project-related data to the Google Cloud environment. While transferring data to the cloud, the key things to focus on are reliability, security, scalability, and the ease of managing the transfer process. With these points in mind, Google Cloud provides four major data transfer utilities to meet customer requirements across a variety of use cases. In general, these utilities are useful for any kind of data transfer purposes, including data center migration, data backup, content storage, and machine learning. As our current focus is on making data available for machine learning use cases, we can utilize any of the following transfer solutions:

• Google Cloud Storage Transfer tools
• BigQuery Data Transfer Service
• Storage Transfer Service
• Transfer Appliance

Let’s understand each of these transfer solutions.

Google...

Where to store data

GCS and BQ are two recommended options for storing any machine learning use case-related datasets for high security and efficiency purposes. If the underlying data is structured or semi-structured, BQ is the recommended option due to its off-the-shelf features for manipulating or processing structured datasets. If the data contains images, videos, audio, and unstructured data, then GCS is the suitable option to store it. Let’s learn about these two data storage systems in more detail.

GCS – object storage

A significant amount of data that we collect from real-world applications is in unstructured form. Some examples are images, videos, emails, audio files, web pages, and sensor data. Managing and storing such huge amounts of unstructured data affordably and efficiently is quite challenging. Nowadays, object storage has become a preferable solution for storing such large amounts of static data and backups. Object storage is a computer data architecture...

Transforming data

Raw data present in real-world applications is often unstructured and noisy. Thus, it cannot be fed directly to machine learning algorithms. We often need to apply several transformations on raw data and convert it into a format that is well supported by machine learning algorithms. In this section, we will learn about multiple options for transforming data in a scalable and efficient way on Google Cloud.

Here are three common options for data transformation in the GCP environment:

• Ad hoc transformation within Jupyter Notebooks
• Cloud Data Fusion
• Dataflow pipelines for scalable data transformations

Let’s learn about these three methods in more detail.

Ad hoc transformations within Jupyter Notebook

Machine learning algorithms are mathematical and can only understand numeric data. For example, in computer vision problems, images are converted into numerical pixel values before they’re fed into a model. Similarly, in the...

Summary

Managing data effectively is really important for saving time, cost, and complexity for every organization. A machine learning practitioner should be aware of the best options for transferring, storing, and transforming data to build machine learning solutions more efficiently. In this chapter, we learned about multiple ways of bringing data into the Google Cloud environment. We discussed the best options for storing it based on the characteristics of the data. Finally, we discussed multiple different tools and methods for transforming/processing data in a scalable manner.

After reading this chapter, you should feel confident about choosing the best option for moving or transferring data into your Google Cloud environment based on the requirements of the use case. Choosing the best place to store data and the best strategy to analyze and transform data should be easier as we now know the pros and cons of different options. In the next chapter, we will deep dive into Vertex...