Chapter 11: Monitoring

In the previous chapter, you learned how to build and manage your data pipeline with AWS Glue in detail. With that knowledge, you are now able to build a data platform powered by AWS Glue. Cool! But this is not the end of your work with the data platform. It is just the starting point.

Imagine that you have built your data platform using AWS Glue. If your data platform does not meet the predefined business requirements, end users will be confused and won’t be able to make a reasonable decision based on the data. If your data platform gives outdated results, the decisions made based on the data will also be outdated. If your data platform is too slow, end users won’t be able to make timely decisions and could lose business opportunities. If your data platform does not check data quality and accuracy, no one can use it for critical decisions. If no end users query your data platform due to a lack of knowledge, your data platform is meaningless...

Defining an SLA for a data platform

When operating a data platform, it is essential to define a healthy state for the entire data platform and maintain that state. Think about what kind of state the data platform should be in. It would be good to define an SLA as an indicator of health. This SLA does not always need to be communicated to end users but is used as an internal indicator to measure whether your data platform is healthy or not.

The basic strategy is to maintain a certain data platform state where the SLA is met and then recover to the normal state when it fails. In other words, monitoring is performed to understand when the platform has deviated from a normal state to an abnormal state, and recovery is performed to return the data platform from an abnormal state to a normal state, as illustrated in the following diagram:

Figure 11.1 – The monitoring cycle

Now, I would like to look at an example of how to define the health of a data platform...

Monitoring the SLA of a data platform

Let’s think about the implementation of a mechanism to monitor the health of a data platform. There are two common strategies to identify the state of a data platform:

• Fact-based approach: Inspect the end user activities and retrieve the metrics.
• Simulation-based approach: Simulate the end user activities and measure the metrics.

To monitor performance and cost SLAs, you can inspect the end user activities from the metrics and log messages. For Amazon Athena, you will see a variety of metrics including query planning time and total execution time via Amazon CloudWatch (https://docs.aws.amazon.com/athena/latest/ug/query-metrics-viewing.html) or Amazon Athena’s query history (https://docs.aws.amazon.com/athena/latest/ug/querying.html#queries-viewing-history). For Amazon Redshift, you can rely on system tables: SVL_QUERY_SUMMARY (https://docs.aws.amazon.com/redshift/latest/dg/using-SVL-Query-Summary.html) and SVL_QUERY_REPORT...

Monitoring the components of a data platform

Data platforms can consist of multiple components: data ingestion jobs, ETL jobs, data crawlers, data catalogs, ad hoc query engines, BI dashboards, and more. In order to detect potential issues that can affect an end user’s experience, it is recommended that you monitor the individual components of your data platform. Here’s a list of key topics to monitor AWS Glue and its related components:

• Monitoring overall statistics
• Monitoring state changes
• Monitoring delay
• Monitoring performance
• Monitoring common failures
• Monitoring log messages

In the following sub-sections, we will look at each of these key topics in detail.

Monitoring overall statistics

For AWS Glue jobs, Glue Studio gives you an aggregated view of the overall statistics, as shown in the following screenshot. This is useful for monitoring the trends of an entire AWS account/region:

Figure 11.3 –...

Analyzing usage

Due to the nature of a data platform, it is not practical to build it once and leave it as it is without any updates. This is because data volume, velocity, and variety increase day by day. Also, how the data is consumed and utilized can often vary. It is practical to build a platform based on the minimum requirement, start using it, measure end user activities, and continuously improve it based on end user feedback.

After you release the data platform to end users, you might see issues such as the following:

• Less usage than expected
• Less adoption in specific teams
• Too many escalations from end users

To make the data platform useful for your end users, you need to maintain and keep improving the platform by tracking and analyzing end user activities.

Let’s look at how user activity can be measured for each type of activity. For example, if it is a simple data reference, it can be recorded and measured in the Amazon S3 server access...

Summary

In this chapter, you learned how to monitor your entire data platform and your AWS Glue components and related services. Additionally, you learned how to analyze end user activities. Monitoring is essential to keep an SLA and also continuously improve a data platform. Now you should be able to define a reasonable SLA based on the requirement and implement a mechanism to monitor your data platform efficiently.

In the next chapter, you will learn more details about how to tune, debug, and troubleshoot issues when using AWS Glue.