The AWS Data Engineer’s Toolkit

Back in 2006, Amazon launched Amazon Web Services (AWS) to offer on-demand delivery of IT resources over the internet, essentially creating the cloud computing industry. Ever since then, AWS has innovated at an incredible pace, continually launching new services and features to offer broad and deep functionality across a wide range of IT services.

Traditionally, organizations built their own big data processing systems in their data centers, implementing commercial or open-source solutions designed to help them make sense of ever-increasing quantities of data. However, these systems were often complex to install, requiring a team of people to maintain, optimize, and update, and scaling these systems was a challenge, requiring significant expenditure on infrastructure and delays while waiting for hardware vendors to install new compute and storage systems.

Cloud computing has enabled the removal of many of these challenges, including the...

Technical requirements

You can find the code files of this chapter in the GitHub repository at the following link: https://github.com/PacktPublishing/Data-Engineering-with-AWS-2nd-edition/tree/main/Chapter03.

Join our book community on Discord

https://packt.link/EarlyAccessCommunity

Back in 2006, Amazon launched Amazon Web Services (AWS) to offer on-demand delivery of IT resources over the internet, essentially creating the cloud computing industry. Ever since then, AWS has been innovating at an incredible pace, continually launching new services and features to offer broad and deep functionality across a wide range of IT services.Traditionally organizations built their own big data processing systems in their data centers, implementing commercial or open source solutions designed to help them make sense of ever-increasing quantities of data. However, these systems were often complex to install, requiring a team of people to maintain, optimize, and update, and scaling these systems was a challenge, requiring large infrastructure spend and significant delays while waiting for hardware vendors to install new compute and storage systems.Cloud computing has enabled the removal of many of these...

Technical requirements

You can find the code files of this chapter in the GitHub repository using the following link: https://github.com/PacktPublishing/Data-Engineering-with-AWS-2nd-edition/ tree/main/Chapter03

AWS services for ingesting data

The first step in building big data analytic solutions is to ingest data from a variety of sources into AWS. In this section, we introduce some of the core AWS services designed to help with this; however, this should not be considered a comprehensive review of every possible way to ingest data into AWS.Don't feel overwhelmed by the number of services we cover in this section! We will explore approaches for deciding on the right service for your specific use case in later chapters, but it is important to have a good understanding of the available tools upfront.

Overview of Amazon Database Migration Service (DMS)

One of the most common ingestion use cases is to sync data from a database system into an analytic pipeline, either landing the data in an Amazon S3-based data lake, or in a data warehousing system such as Amazon Redshift.AWS Database Migration Service (DMS) is a versatile tool that can be used to migrate an existing database system to a new...

AWS services for transforming data

Once your data is ingested into an appropriate AWS service, such as Amazon S3, the next stage of the pipeline needs to transform the data to optimize it for analytics and to make it available to your data consumers.Some of the tools we discussed in the previous section for ingesting data into AWS can perform light transformations as part of the ingestion process. For example, Amazon DMS can write out data in Parquet format (a format optimized for analytics), as can Kinesis Firehose. However, heavier transformations are often required to fully optimize your data for a differing set of analytic tasks and diverse data consumers, and in this section, we will examine some of the core AWS services that can be used for this.

Overview of AWS Lambda for light transformations

AWS Lambda provides a serverless environment for executing code, and is one of AWS's most popular services. You can trigger your Lambda function to execute your code in multiple ways...

AWS services for orchestrating big data pipelines

As discussed in Chapter 2, Data Management Architectures for Analytics, a data pipeline can be built to bring in data from source systems, and then transform that data, often moving the data through multiple stages, further transforming or enriching the data as it moves through each stage.An organization will often have tens or hundreds of pipelines that work independently or in conjunction with each other on different datasets and perform different types of transformations. Each pipeline may use multiple services to achieve the goals of the pipeline, and orchestrating all the varying services and pipelines can be complex. In this section, we look at a number of AWS services that help with this orchestration task.

Overview of AWS Glue workflows for orchestrating Glue components

In the AWS services for transforming data section, we covered AWS Glue, a service that includes a number of components. As a reminder, they are as follows:

• A...

AWS services for consuming data

Once the data has been transformed and optimized for analytics, the various data consumers in an organization need easy access to the data via a number of different types of interfaces. Data scientists may want to use standard SQL queries to query the data, while data analysts may want to both query the data in place using SQL and also load subsets of the data into a high-performance data warehouse for low-latency, high-concurrency queries and scheduled reporting. Business users may prefer accessing data via a visualization tool that enables them to view data represented as graphs, charts, and other types of visuals.In this section, we introduce a number of AWS services that enable different types of data consumers to work with our optimized datasets. We don't cover all services that can be used to consume data in this section, but instead highlight the primary services relevant to the data engineering role.

Overview of Amazon Athena for SQL queries...