Chapter 11: Operational Excellence – Monitoring, Optimization, and Troubleshooting

In this chapter, we will focus on operational excellence. Operational excellence in this chapter has three components: monitoring Athena to ensure it is healthy and running normally, optimizing our usage of the system for cost and performance, and, lastly, how to troubleshoot issues when they occur.

When monitoring systems, it is essential to know what to monitor and what steps to take when something goes wrong. This information is valuable because when the system is not operating correctly, the data will give you clues on possible issues, which reduces investigation time. You can also act before problems occur, preventing calls from users on why things are not working. We will look into processes that can be put in place to ensure that Athena and our usage of it are normal and efficient. When there are issues, we will know how to fix common problems.

We also want to get the most out of...

Technical requirements

For this chapter, if you wish to follow some of the walk-throughs, you will require the following:

• Internet access to GitHub, S3, and the AWS Management Console.
• A computer with a Chrome, Safari, or Microsoft Edge browser installed.
• An AWS account and accompanying IAM user (or role) with sufficient privileges to complete this chapter's activities. For simplicity, you can always run through these exercises with a user that has full access. However, we recommend using scoped-down IAM policies to avoid making costly mistakes and to learn how to best use IAM to secure your applications and data. You can find a minimally scoped IAM policy for this chapter in the book's accompanying GitHub repository, listed as chapter_11/iam_policy_chapter_11.json (https://bit.ly/3hgOdfG). This policy includes the following:
  • Permissions to read, list, and write access to an S3 bucket
  • Permissions to read and write access to the AWS Glue Data Catalog databases...