Pragmatic Data Processing and Analysis

Data needs to be analyzed, transformed, and processed first before using it when training machine learning (ML) models. In the past, data scientists and ML practitioners had to write custom code from scratch using a variety of libraries, frameworks, and tools (such as pandas and PySpark) to perform the needed analysis and processing work. The custom code prepared by these professionals often needed tweaking since different variations of the steps programmed in the data processing scripts had to be tested on the data before being used for model training. This takes up a significant portion of an ML practitioner’s time, and since this is a manual process, it is usually error-prone as well.

One of the more practical ways to process and analyze data involves the usage of no-code or low-code tools when loading, cleaning, analyzing, and transforming the raw data from different data sources. Using these types of tools will significantly speed...

Technical requirements

Before we start, it is important that we have the following ready:

• A web browser (preferably Chrome or Firefox)
• Access to the AWS account used in the first four chapters of the book

The Jupyter notebooks, source code, and other files used for each chapter are available in this repository: https://github.com/PacktPublishing/Machine-Learning-Engineering-on-AWS.

Important Note

Make sure to sign out and NOT use the IAM user created in Chapter 4, Serverless Data Management on AWS. In this chapter, you should use the root account or a new IAM user with a set of permissions to create and manage the AWS Glue DataBrew, Amazon S3, AWS CloudShell, and Amazon SageMaker resources. It is recommended to use an IAM user with limited permissions instead of the root account when running the examples in this book. We will discuss this along with other security best practices in further detail in Chapter 9, Security, Governance, and Compliance Strategies...

Getting started with data processing and analysis

In the previous chapter, we utilized a data warehouse and a data lake to store, manage, and query our data. Data stored in these data sources generally must undergo a series of data processing and data transformation steps similar to those shown in Figure 5.1 before it can be used as a training dataset for ML experiments:

Figure 5.1 – Data processing and analysis

In Figure 5.1, we can see that these data processing steps may involve merging different datasets, along with cleaning, converting, analyzing, and transforming the data using a variety of options and techniques. In practice, data scientists and ML engineers generally spend a lot of hours cleaning the data and getting it ready for use in ML experiments. Some professionals may be used to writing and running custom Python or R scripts to perform this work. However, it may be more practical to use no-code or low-code solutions such as AWS Glue DataBrew...

Preparing the essential prerequisites

In this section, we will ensure that the following prerequisites are ready before proceeding with the hands-on solutions of this chapter:

• The Parquet file to be analyzed and processed
• The S3 bucket where the Parquet file will be uploaded

Downloading the Parquet file

In this chapter, we will work with a similar bookings dataset as the one used in previous chapters. However, the source data is stored in a Parquet file this time, and we have modified some of the rows so that the dataset will have dirty data. That said, let’s download the synthetic.bookings.dirty.parquet file onto our local machine.

You can find it here: https://github.com/PacktPublishing/Machine-Learning-Engineering-on-AWS/raw/main/chapter05/synthetic.bookings.dirty.parquet.

Note

Note that storing data using the Parquet format is preferable to storing data using the CSV format. Once you need to work with much larger datasets, the difference...

Automating data preparation and analysis with AWS Glue DataBrew

AWS Glue DataBrew is a no-code data preparation service built to help data scientists and ML engineers clean, prepare, and transform data. Similar to the services we used in Chapter 4, Serverless Data Management on AWS, Glue DataBrew is serverless as well. This means that we won’t need to worry about infrastructure management when using this service to perform data preparation, transformation, and analysis.

Figure 5.2 – The core concepts in AWS Glue DataBrew

In Figure 5.2, we can see that there are different concepts and resources involved when using AWS Glue DataBrew. We need to have a good idea of what these are before using the service. Here is a quick overview of the concepts and terms used:

• Dataset – Data stored in an existing data source (for example, Amazon S3, Amazon Redshift, or Amazon RDS) or uploaded from the local machine to an S3 bucket.
• Recipe –...

Preparing ML data with Amazon SageMaker Data Wrangler

Amazon SageMaker has a lot of capabilities and features to assist data scientists and ML engineers with the different ML requirements. One of the capabilities of SageMaker focused on accelerating data preparation and data analysis is SageMaker Data Wrangler:

Figure 5.18 – The primary functionalities available in SageMaker Data Wrangler

In Figure 5.18, we can see what we can do with our data when using SageMaker Data Wrangler:

1. First, we can import data from a variety of data sources such as Amazon S3, Amazon Athena, and Amazon Redshift.
2. Next, we can create a data flow and transform the data using a variety of data formatting and data transformation options. We can also analyze and visualize the data using both inbuilt and custom options in just a few clicks.
3. Finally, we can automate the data preparation workflows by exporting one or more of the transformations configured in the...

Summary

Data needs to be cleaned, analyzed, and prepared before it is used to train ML models. Since it takes time and effort to work on these types of requirements, it is recommended to use no-code or low-code solutions such as AWS Glue DataBrew and Amazon SageMaker Data Wrangler when analyzing and processing our data. In this chapter, we were able to use these two services to analyze and process our sample dataset. Starting with a sample “dirty” dataset, we performed a variety of transformations and operations, which included (1) profiling and analyzing the data, (2) filtering out rows containing invalid data, (3) creating a new column from an existing one, (4) exporting the results into an output location, and (5) verifying whether the transformations have been applied to the output file.

In the next chapter, we will take a closer look at Amazon SageMaker and we will dive deeper into how we can use this managed service when performing machine learning experiments...

Further reading

For more information on the topics covered in this chapter, feel free to check out the following resources:

• AWS Glue DataBrew product and service integrations (https://docs.aws.amazon.com/databrew/latest/dg/databrew-integrations.html)
• Security in AWS Glue DataBrew (https://docs.aws.amazon.com/databrew/latest/dg/security.html)
• Create and Use a Data Wrangler Flow (https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-data-flow.html)
• Data Wrangler – Transform (https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-transform.html)
• Data Wrangler – Troubleshooting (https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-trouble-shooting.html)