Interactive Analytics

In this chapter, we will look at the following key topics:

• Analytics using Amazon Athena
• Analytics using Presto, Trino, and Hive on Amazon EMR

One of the fundamental principles of building a modern data architecture on AWS is hinged around using purpose-built tools for solving specific use cases. An enterprise data platform once fully built has many components, each with a specific purpose for solving a particular business use case.

In Chapter 2, Scalable Data Lakes, we went through the fundamentals of building a data lake on AWS using Amazon S3 as the storage layer and the AWS Glue Data Catalog as the technical metadata layer. Each layer of the data lake has data that may be of use to different personas in an organization. The most basic ask from each of these personas will be to provide them the ability to query datasets in the data lake using the SQL syntax so that they can derive insights from the data. Interactive analytics, using specific...

Analytics using Amazon Athena

Amazon Athena is a serverless service that allows anyone to interactively query data in an S3 data lake using American National Standards Institute (ANSI) SQL. Athena is integrated with the Glue Data Catalog, which means that as soon as the metadata about the data is captured in the catalog, Athena is able to view and query those tables without any additional setup.

With no infrastructure to manage, along with direct out-of-the-box integration with Glue Data Catalog, this makes Athena a very popular service for multiple personas in the organization for interactive and ad hoc analytics. And being a serverless service, the pricing model of the service, 5 dollars per terabyte of data scanned, is also very appealing to organizations. With the correct optimization techniques, you can effectively roll out Athena as the service of choice throughout your organization for all types of interactive analytics use cases, for querying the data stored in the S3 data...

Analytics using Presto, Trino, and Hive on Amazon EMR

If you recall from Chapter 5, we introduced Amazon EMR as one of the services for processing big data. EMR has over 25 open source projects, and we went through a use case where Apache Spark in EMR was leveraged to solve a data processing problem. EMR also has a few projects that assist in ad hoc query execution and allow users to interactively executive SQL queries to get the data stored in the S3 data lake. Let’s shed some light on these projects.

Presto/Trino

Presto is an open source project that provides a fast analytics query execution engine for data stored in many types of storage, most commonly used with data stored in data lakes. Presto, also known as PrestoDB, was first created on Facebook. In 2019, Presto development eventually forked into two, with PrestoDB and PrestoSQL. To keep the name confusion at a minimum, PrestoSQL was renamed Trino in 2020.

Amazon EMR supports both PrestoDB and Trino. Organizations...

Summary

In this chapter, we looked at how Amazon Athena and Presto, Trino, and Hive on EMR help organizations perform ad hoc interactive data analytics on the data stored in the S3 data lake. Athena is a serverless platform that integrates with the Glue Data Catalog and provides data analysts with the ability to write and execute SQL queries without having to manage the platform itself. Using Athena, organizations can focus on the business logic needed for reports versus spending time on creating and managing the infrastructure that’s required by the platform.

We also looked at cases when creating a Presto/Trino cluster on Amazon EMR may be more beneficial for interactive analytics. This is particularly helpful when there are very large volumes of datasets that need to be scanned by thousands of queries on a daily basis and where performance SLAs are strict. Using Presto/Trino on EMR, customers can control cost and at the same time improve query performance by custom tuning...

References

• Amazon Athena workshop: https://catalog.us-east-1.prod.workshops.aws/workshops/9981f1a1-abdc-49b5-8387-cb01d238bb78/en-US