Preface

These days, organizations have gravitated toward data-driven business. Today, data integration across various data sources has become a key driver for businesses. In the cloud, data integration services such as AWS Glue do the undifferentiated heavy lifting based on serverless infrastructure. AWS Glue helps you to integrate data across different sources and build a data lake at scale in a serverless fashion without maintaining infrastructure.

This book shows you how AWS Glue can be used to solve real-world problems, along with teaching you about data processing, data integration, and building data lakes. It allows you to learn how to perform various aspects of data integration techniques such as data ingestion from various sources, data layout optimization, data and metadata management, and data pipeline management. Further, it covers data analysis use cases such as ad hoc queries, visualization, and real-time analysis using AWS Glue. Additional topics such as CI/CD, data quality validation, data sharing, and data security aspects, such as access control, encryption, auditing, and networking, are also covered. Toward the end, the book focuses on providing various monitoring options and the best practices for tuning, debugging, and troubleshooting.

The book takes you through the AWS Glue features such as jobs, the Data Catalog, crawlers, DataBrew, Glue Studio, custom connectors, and so on, in addition to AWS Lake Formation.

By the end of this book, you will be able to integrate data across different sources and build a data platform for scalable analysis using AWS Glue.

Who this book is for

This book is designed for data engineers, ETL developers, and data analysts who want to understand how AWS Glue can help to solve their business problems. Basic knowledge of AWS data services is assumed. Experience with AWS Glue is also preferred but not required. Even without prior knowledge, you can start learning AWS Glue with the book. Most of the features are accompanied by a walkthrough to help you understand the concepts that are explained in each chapter.

What this book covers

Chapter 1, Data Management – Introduction and Concepts, introduces basic concepts associated with data management.

Chapter 2, Introduction to Important AWS Glue Features, introduces some important AWS Glue features.

Chapter 3, Data Ingestion, describes how to ingest data across multiple data stores.

Chapter 4, Data Preparation, describes typical data preparation use cases with both a GUI-based approach and a source code-based approach using AWS Glue.

Chapter 5, Designing Data Layouts, describes how to optimize data layout on Amazon S3 using AWS Glue.

Chapter 6, Data Management, describes how to manage, clean up, and enrich data using AWS Glue.

Chapter 7, Metadata Management, describes how to populate and maintain metadata based on data using AWS Glue.

Chapter 8, Data Security, describes how to secure your data by access control, encryption, auditing, and network security using AWS Glue.

Chapter 9, Data Sharing, describes how to share your data across multiple accounts to democratize your data lake.

Chapter 10, Data Pipeline Management, describes how to build and orchestrate a data-processing pipeline using AWS Glue.

Chapter 11, Monitoring, describes how to monitor a data lake and AWS Glue components.

Chapter 12, Tuning, Debugging, and Troubleshooting, describes the best practices to tune, debug, and troubleshoot typical use cases.

Chapter 13, Data Analysis, describes common options to analyze data using AWS analytics services.

Chapter 14, Machine Learning Integration, describes how to utilize your data for a machine learning workload.

Chapter 15, Architecting Data Lakes for Real-World Scenarios and Edge Cases, describes end-to-end examples of architecting data lakes.

To get the most out of this book

All walkthroughs will require a web browser (Google Chrome, Mozilla Firefox, Microsoft Edge, or Safari) installed on a computer in order to use AWS Management Console, and you’ll need an AWS account to access the AWS Console and utilize AWS resources. Next to that, you’ll need to install the AWS Command Line Interface (AWS CLI) on a computer to run commands:

Not all the chapters’ walkthroughs require an AWS CLI installation. You’ll be informed in each chapter when you need further requirements.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Serverless-ETL-and-Analytics-with-AWS-Glue. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/fTqGe.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “We used the glueContext.write_dynamic_frame.from_options() method to write the data to Amazon S3.”

A block of code is set as follows:

root

 |-- ColumnA: string (nullable = true)

 |-- ColumnB: string (nullable = true)

Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “This can be done by navigating to AWS Glue Studio console | Connectors | Marketplace Connectors and subscribing to Cloudwatch Metrics connector for AWS Glue.”

Tips or Important Notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [e-mail] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [e-mail] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read Serverless ETL and Analytics with AWS Glue, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.