Chapter 5. Monitoring Amazon EC2 Instances

In this chapter, we will cover recipes for:

• Collecting EC2 metrics using AWS CloudWatch

• Collecting custom metrics from EC2 instances

• Monitoring costs using CloudWatch

• Sending an e-mail based on a CloudWatch alarm

• Using CloudWatch Logs

Introduction

Amazon CloudWatch is AWS's monitoring service. It can be used to monitor your applications and AWS resources. AWS CloudWatch works with all key AWS services including EC2, RDS, DynamoDB, Elastic MapReduce, Kinesis, CloudSearch, and so on. CloudWatch provides several out-of-the-box metrics such as CPU utilization, network utilization, and disk I/O metrics from EC2 instances. However, you can also collect custom metrics from your applications.

AWS Management Console can be used to view, search, and graph the metrics data loaded by Amazon CloudWatch. CloudWatch stores data as a series of time stamped data points.

One of most common use cases for CloudWatch is to use it to set alarms and take actions, based on the metrics. For example, you can stop, terminate, recover, or reboot an EC2 instance in response to a CloudWatch alarm being raised. CloudWatch Logs lets you collect your application, system, and custom log files into AWS CloudWatch Logs for analysis.

AWS CloudWatch dashboard...

Collecting EC2 metrics using AWS CloudWatch

You can collect basic metrics such as CPU utilization, network I/O, and disk I/O metrics from your EC2 instances. There are two types of CloudWatch monitoring services—basic and detailed. In basic monitoring, the Amazon EC2 metrics data is collected at 5 minute periods and retained for 2 weeks, at no charge. The metrics are preselected and limited in number. Basic monitoring is automatically enabled for all EC2 instances. In detailed monitoring, the metrics are collected at 1-minute intervals and charged per hour per instance.

Regardless of the monitoring type, the metrics are aggregated by autoscaling group and EBS (if you are using autoscaling or EBS). If you want detailed monitoring, you can enable it, both while creating a new EC2 instance and for existing EC2 instances. You can access the metrics data using either the CloudWatch API or the AWS Management Console.

How to do it…

1. Enable detail monitoring when launching an instance.

   Execute the following...

Collecting custom metrics from EC2 instances

You can collect custom metrics from your applications, for example, the number of active sessions, response latency, and many more. More importantly, you can report custom metrics in application and business terms. For example, average number of orders processed per minute, today, on an e-commerce site. These business metrics can help you with capacity planning and allocating suitable budgets for your cloud infrastructure based on the business impact.

Custom metrics help you monitor your applications, directly, from CloudWatch. Using these custom metrics, you can create alarms that can, for example, add instances in the autoscaling group. There is a simple PUT API call to collect custom metrics. AWS also provides monitoring scripts for Linux and Windows that send custom metrics to the AWS CloudWatch.

How to do it…

1. Installing AWS Java SDK.

   It helps you access AWS CloudWatch service from Java applications. In your Maven dependency section, add the following...

Monitoring costs using CloudWatch

AWS CloudWatch can help you track and manage your cloud costs. This is especially useful where your AWS resource usage varies significantly and you need to log in to the AWS portal several times a day to track the usage. You can set appropriate alarms when the charges exceed a certain predefined threshold. To enable billing alerts for your AWS account, you have to log in as the account owner and edit the configuration settings. Hence, you can send notifications based on these metrics as e-mails and/or route them to other applications for further processing.

The estimated charges are calculated and sent several times a day to CloudWatch. These metrics include estimated total charges and charges by service. In addition, if you are using the consolidated billing option, then your CloudWatch metrics for estimated charges by linked account and estimated charges by linked account and service are also stored. This data is stored for 2 weeks, and it includes the...

Sending an e-mail based on a CloudWatch alarm

This recipe is similar to the previous one. However, in this case, we set up metrics, alarms, and notifications that are vital to the operations team. CloudWatch alarms help you notify users by e-mail, if any of the specified metrics cross their predefined thresholds. For example, you can collect latency metrics from the ELB and trigger an alarm that sends an e-mail notification to the operations team, if the latency exceeds 60 milliseconds. CloudWatch uses Simple Notification Service (SNS) to send these e-mails. The administrators and operations teams subscribe to this SNS topic to receive the CloudWatch notifications.

How to do it…

1. Create an SNS topic.

   In the following example, the command creates the SNS topic called EC2CpuHighTopic. Record TopicArn for further usage.

   $ aws sns create-topic 
   --name EC2CpuHighTopic
   

2. Subscribe to the SNS topic.

   Execute the following command to specify the e-mail ID <[e-mail]> subscribed to the SNS...

Using CloudWatch Logs

AWS CloudWatch Logs helps you store, monitor, and analyze your application, system and custom logs, centrally. Using CloudWatch Logs you can monitor your logs in near real time for specific errors and exceptions in your application. For example, you might want to monitor exceptions such as NullPointerException, ArrayIndexOutOfBounds, and ArithmeticException in your Java application.

We will need to aggregate the logs from multiple hosts in an environment where instances are added and deleted, dynamically. AWS CloudWatch is not restricted to work with EC2 instances only, it can also be used for on-premise servers and servers hosted on other public clouds. You can analyze your logs and archive them for access later.

You can extract metrics from logs as they come into AWS CloudWatch using the metric filter. For example, you may want to monitor your web server logs for 4xx and 5xx status codes or monitor your server's log files for OS specific error conditions. In addition...