Chapter 6: AWS Glue and AWS Lake Formation

Although this book focuses on Athena and its rich functionality, you should be aware of AWS Glue and AWS Lake Formation. These services can be used with Athena to implement use cases that Athena cannot alone. AWS Lake Formation was created to help customers simplify creating data lakes by providing tools to help ingest data, secure data, and reduce the time it takes to get a functional data lake. Lake Formation is a layer that exists on top of AWS Glue and uses Glue's components as building blocks.

One of the main features that Lake Formation brings is fine-grained access controls and auditing to several AWS services, including Athena. Lake Formation augments AWS IAM to help secure the data lake. IAM provides authentication of the user, while Lake Formation provides authorization based on the principle that is requesting data. Every authorization request that goes through Lake Formation generates audit events in CloudTrail that are...

Technical requirements

For this chapter, if you wish to follow some of the walkthroughs, you will need the following:

• Internet access to GitHub, S3, and the AWS Console.
• A computer with either Chrome, Safari, or Microsoft Edge installed on it.
• An AWS account and accompanying IAM user (or role) with sufficient privileges to complete this chapter's activities. For simplicity, you can always run through these exercises with a user that has full access. However, we recommend using scoped-down IAM policies to avoid making costly mistakes and learn how to best use IAM to secure your applications and data. You can find a minimally scoped IAM policy for this chapter in this book's accompanying GitHub repository, which is listed as chapter_6/iam_policy_chapter_6.json. This policy includes the following:
  • Permissions to create and list IAM roles and policies:
    • We will be creating a service role for an AWS Glue Crawler to assume.
  • Permissions to read, list, and write access...

Summary

In this chapter, you learned what AWS Glue and AWS Lake Formation provide when building and maintaining data lakes on AWS. We then focused on Lake Formation's ability to provide fine-grained access controls and the benefits and limitations of this. We also went through a sample process of enabling Lake Formation access controls for a new database and how it works within Athena. Lastly, we touched on Lake Formation governed tables, what they are, and how they can solve many issues with storing datasets on a distributed filesystem. There are more advanced features of Lake Formation, and we will dive deeper into governed tables in Chapter 14, Lake Formation – Advanced Topics.

In the next part of this book, we will get our hands dirty by using Amazon Athena in various settings ranging from ad hoc data analysis, using Athena to build ETL pipelines, and building applications that use Athena. We'll also take some time to cover how you can troubleshoot and tune...

Further reading

To learn more about the topics that were covered in this chapter, take a look at the following resources:

• AWS Lake Formation resources, including blog posts and demo videos: http://amzn.to/394z9x7
• Registering an encrypted Amazon S3 location – AWS Lake Formation: https://amzn.to/3hf39uW
• Registering an Amazon S3 location in another AWS account – AWS Lake Formation: https://amzn.to/3baTVfI
• Limitations of using Lake Formation security with Athena: https://amzn.to/3nwAvGN