Chapter 13: CloudWatch and Logging

Monitoring and logging your databases using CloudWatch is a key topic in the exam, and also a technique required by all database administrators. We have previously learned about CloudWatch and logging at a high level, but in this chapter, we are going to learn some more advanced features and options within CloudWatch. We will end the chapter with a hands-on lab, where we will customize our monitoring and alerts, and generate some heavy load on an Relational Database Service (RDS) instance to ensure we receive an alert. We will also look at a tool called Application Insights that allows you to monitor your databases as part of a wider application stack to help identify the root cause of any incident or outage.

In this chapter, we're going to cover the following main topics:

• Overview of CloudWatch and logging
• Working with CloudWatch
• Understanding Performance Insights
• Understanding RDS Enhanced Monitoring
• Configuring CloudWatch...

Technical requirements

You will require an AWS account with root access; not everything we will do in this chapter will be available in the Free Tier, which means you may incur a small cost to follow the hands-on sections. You will also require AWS command-line interface (CLI) access. This AWS guide (https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) explains the steps required, but I will summarize them here: 

1. Open an AWS account, if you have not already done so. 
2. Download the AWS CLI latest version from here: https://docs.aws.amazon.com/cli/latest/userguide/welcome-versions.html#welcome-versions-v2.
3. Create an admin user: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html 
4. Create an access key for your administration user: https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html#getting-started_create-admin-group-cli.
...

Overview of CloudWatch and logging

CloudWatch is the primary monitoring and logging service offered by AWS. It acts as a data and metrics repository storing all of your RDS monitoring metrics in near real time. CloudWatch can generate graphs, allow you to download raw values into several different formats, trigger alarms and alerts, call AWS Lambda functions, and allow you to read log files from multiple databases at once.

CloudWatch metrics can be viewed graphically, both via the RDS and CloudWatch dashboards. It also offers full Application Programming Interface (API) and AWS CLI integration, allowing you to use other tools, such as Amazon QuickSight, to view and analyze the data as required. The metrics you can view from the RDS dashboard are more limited but offer a clear and quick overview of your database health. The following figure shows some of the metrics available via the RDS dashboard:

Figure 13.1 – RDS dashboard CloudWatch metrics

You...

Working with CloudWatch

CloudWatch is the central monitoring and alerting repository for all AWS services. As such, it is extremely powerful and features many advanced tools and processes, which we do not need to know for the Database Specialty exam. However, given the usefulness of many of these additional features, there is a recommended book in the Further reading section for anyone wanting to know more beyond the scope of this chapter.

CloudWatch can be used to monitor any of your databases running within AWS, not just RDS. It can also be used to monitor databases running on-premises by installing the CloudWatch agent within your data center.

The main page of CloudWatch is Dashboards. You can create your own dashboards to show the metrics that are the most useful to you across all AWS services. For example, if you had an application running on EC2 with a load balancer and an RDS instance, you could create a dashboard that monitored the CPU and memory of the EC2 instances...

Understanding Performance Insights

Performance Insights allows you to view queries and database load in real time through a graphical interface. It is available for all RDS database engines, but is not available on the t3.small and lower instance classes.

Performance Insights is accessed directly from the RDS dashboard rather than via CloudWatch. It is not a standard feature and it must be enabled for each instance that you need. This can be done at instance provisioning or later, by modifying the instance and enabling Performance Insights, which will not incur any outage. Performance Insights is free for data stored for a maximum of 7 days, and it is chargeable if you need to keep data for a longer period.

The Performance Insights dashboard offers an overview graph that you can customize with any metrics you wish, and further down the page, you can see details for specific queries and database load. The following figure shows how you can add multiple metrics to the graph to...

Understanding RDS Enhanced Monitoring

CloudWatch offers limited monitoring of the virtual machine or host that your RDS instances run on. To get a deeper and more accurate view, you need to enable RDS Enhanced Monitoring. This installs a small agent onto the RDS host that obtains and sends OS metrics back to CloudWatch up to every 1 second.

You can use these metrics in conjunction with database-level monitoring to get a highly accurate view of the real-time workload of your database. This can be very useful if you need to decide to increase an instance class to overcome a performance issue. By using Enhanced Monitoring, you can more accurately see whether the bottleneck is at the host level and, therefore, you would benefit from increasing the instance class or not.

Enabling Enhanced Monitoring also allows you to view OS metrics from within the RDS dashboard, which reduces the need for your databases to have access to CloudWatch. The following figure shows some of the metrics...

Configuring CloudWatch monitoring and alerts

We are now going to practice a hands-on session where we will set up some CloudWatch metrics, create a dashboard so we can view all the critical metrics in one place, and finally, configure some alarms to send us emails about database issues.

If you do not have any RDS MySQL databases, then please create one. Make sure you configure the security groups to allow you to log in to it, as we will be connecting and generating traffic to trigger alarms. Make sure you request it to send all logs to CloudWatch.

Let's start by creating a CloudWatch dashboard.

Creating a CloudWatch dashboard

Let's create a CloudWatch dashboard that will allow us to monitor all our key metrics in one place. If you have more than one database, you can monitor them all from one dashboard, but in this lab, we only use one database:

1. Log in to the AWS console and navigate to CloudWatch.
2. Click Dashboard, and then click Create dashboard....

Summary

In this chapter, we learned about how to use CloudWatch to monitor your databases running in AWS. CloudWatch offers a powerful and highly customizable setup of tools, such as dashboards, alarms, and anomaly detection, to finely tune your alerts and monitoring across your database estate.

We learned how to use Enhanced Monitoring to be able to closely monitor the health and performance of the virtual machines that run our RDS databases to help diagnose performance issues more accurately.

Finally, we learned how to use Performance Insights to graphically see ongoing and real-time performance metrics at the query level to help identify the root cause of any database performance issues and areas that could be tuned.

Accurate and comprehensive monitoring of your databases is critical to ensure there are no outages and that performance is maintained, and CloudWatch is one of the best tools to do so. In the exam, there will be a large number of questions about performance...

Cheat sheet

The cheat sheet summarizes the main key points from this chapter:

• CloudWatch is the central monitoring tool for all AWS services.
• RDS, EC2, and other AWS databases all natively send metrics to CloudWatch, and they can also be configured to send their log files to CloudWatch.
• CloudWatch offers Enhanced Monitoring, which lets you accurately monitor the virtual machine the RDS database is running on.
• You can use Performance Insights to get a visual representation of the real-time workload running on your database and its performance to help diagnose performance issues and to find queries that could be tuned.
• CloudWatch allows the creation of dashboards to let you monitor all of your services and key metrics in one place, reducing the need for application teams to have access to the RDS console, which can help improve your database security.
• You can create custom alarms within CloudWatch that can be configured to send emails or text messages if there...

Review

Let's now practice a few exam-style questions:

1. You are a database consultant for a small local company. The company needs to monitor the read and write Input/output Operations Per Second (IOPS) metrics for their AWS MySQL RDS instance and send real-time alerts to their database team. Which AWS services can accomplish this as simply and at as low cost as possible? Choose two answers:
   1. Amazon Simple Email Service (SES)
   2. Amazon CloudWatch
   3. Amazon Simple Queue Service (SQS)
   4. Amazon SNS
   5. AWS Lambda
2. You are a database administrator for an online shopping company. Over the weekend, their critical production database went down due to a larger than expected number of orders. What type of alarms could you create to get a warning of abnormal database load?
   1. Create a CloudWatch alarm based on a static metric, such as database connections or CPU.
   2. Use Performance Insights to observe database load statistics in real time.
   3. Create a custom AWS Lambda function to run a script against the...

Further reading

Infrastructure Monitoring with Amazon CloudWatch: https://www.packtpub.com/product/infrastructure-monitoring-with-amazon-cloudwatch/9781800566057