Logs Generated by AWS Services

As you, your users, and your customers perform your day-to-day functions in AWS, you can capture and store those actions in various logging formats from the different services.

Configuring and enforcing logging across your services helps significantly when identifying potential issues, not just from a security perspective but also from a performance and availability perspective.

This chapter will cover the various types of logs the AWS services produce. You will also examine where they are stored and review their different formats. Knowing how to toggle test logs on and off and how to read the various log files is essential when you need to investigate a particular event or respond to a security incident.

The following main topics will be covered in this chapter:

• S3 access logs
• VPC Flow Logs and traffic monitoring
• Enabling load balancer logs
• Setting up CloudTrail
• Services that publish to CloudWatch Logs

Technical Requirements

Access to the AWS Management Console with an active account and the AWS CLI are required. You also need to have access to a terminal console and a text editor.

S3 Access Logs

Whenever you or your users store or access different objects in Amazon S3, you, as the security professional, may need to know who is accessing the different files, when and where they are accessed, and from what location.

You can capture all the access logs and records of who accessed the various objects in a particular bucket via a simple setting in S3. One caveat is that the access logs for an S3 bucket cannot be stored in the same bucket as the items they are tracking. This means that you need to create a new bucket for storing those logs. You can use a single bucket to track multiple S3 buckets’ access logs. Changing the access policy so that no users besides the security and audit teams or the service role that retrieves the logs may access this bucket is considered best practice.

The access logs are usually pushed to the secondary storage bucket on a best-effort basis, and this can result in a delay of a few hours before delivering the logs to the...

VPC Flow Logs and Traffic Monitoring

You likely have several different public and private subnets within your AWS account allowing external connectivity. You may even have multiple VPCs connected via VPC peering connections or AWS Transit Gateway. Either way, you will have a lot of network traffic traversing your AWS infrastructure from numerous sources, internally and externally, across thousands of interfaces. Flow logs allow you to capture this IP traffic across the network interfaces attached to your resources, which could number in the tens of thousands in a corporate environment.

Flow logs can be configured for the following resources:

• Your VPC
• A subnet within your VPC
• A network interface from your EC2 instances or interfaces created by Elastic Load Balancing (ELB), Amazon RDS, Amazon ElastiCache, Amazon Redshift, Amazon WorkSpaces, NAT gateways, and Transit Gateway

As flow logs can capture information at these levels, they are a tool to help troubleshoot...

Elastic Load Balancer Access Logs

The ELB service allows you to turn on optional logging, which captures and monitors the requests flowing through your load balancers. These logs can also help you analyze traffic patterns and troubleshoot issues with the targets to which the load balancers are sending requests.

The access logs, once enabled, are delivered to an S3 bucket that you specify during the setup and the log files are compressed. You need to decompress the files before accessing the raw log files.

AWS ELB log files can be used for several purposes, including the following:

• Troubleshooting: The ELB log files contain detailed information about the requests that were served by the load balancer, including the source address, the request URL, the HTTP status code, and the response time. This information can be used to troubleshoot issues with your application, such as slow response times, errors, or connectivity problems.
• Performance analysis: ELB log files can...

Services that Publish Logs to CloudWatch Logs

Most services send their logs to the CloudWatch Logs service. This section deals with the different services that generate logs and then send those logs to CloudWatch.

Note

Chapter 8, CloudWatch and CloudWatch Metrics, will discuss in detail how to utilize the capabilities of CloudWatch as a whole, including log retention, log querying, metrics, and events (including Amazon EventBridge).

Although you will not be made to memorize the list of services that can push their logs out to CloudWatch Logs for the exam, it is recommended that you have a robust understanding of which services do not have the ability to send logs to CloudWatch Logs without an intermediary step. Further, knowing the services that send their logs out to CloudWatch Logs (think Lambda functions) still helps you when it comes to reading and understanding the test questions.

One of the best ways to retain this information is not to study these lists but rather...

Logging API Activity with CloudTrail

The service in AWS that enables governance, compliance, risk auditing, and operational auditing is CloudTrail. It does all this by recording the API calls performed either through the AWS Management Console, the AWS CLI, any of the AWS SDKs, or any third-party tool that uses the AWS API. CloudTrail can work in a single account within a single Region, or it can be used to monitor all Regions within that same single account. You can also configure the service to collect events from multiple accounts and then aggregate them in a single bucket for storage. When a company utilizes AWS Organizations often, they oftentimes utilize a specialized account for logging or auditing. Here, the logs from all other accounts in the organization flow into the auditing account, where access is limited except for a select group of individuals.

Figure 7.6: CloudTrail API log flow

To understand how CloudTrail logging works and the information...

Summary

This chapter covered the different types of logs produced by various AWS services and how they can be stored for later use and consumption or, if needed, for an audit.

You saw how S3 can record access to its objects and folders using S3 access logging. You also explored how to troubleshoot and record network activity using VPC Flow Logs. In reviewing another way to capture network traffic, you saw the capabilities of both ELB logging and WAF logs.

You also learned about the service that records all API calls, CloudTrail. You examined how to turn on a new trail for a specific purpose and how to look up events in that trail. In case using the legacy trail becomes limiting, you looked at how to expand the capabilities of CloudTrail using CloudTrail Lake.

Chapter 8, CloudWatch and CloudWatch Metrics, will discuss the CloudWatch service and how it consumes logs. You will also see how CloudWatch can gather and publish predefined and custom metrics from our services.

...

Further Reading

For additional information on the AWS Shared Responsibility Model and an underlying foundation of AWS security, please check out the following resources:

• Logging options for Amazon S3: https://packt.link/7pTi5
• Analyze Network Traffic of Amazon Virtual Private Cloud (VPC) by CIDR blocks: https://packt.link/yiNeO
• Well-Architected Framework – Security Pillar, Detection: https://packt.link/ZLCA7

Exam Readiness Drill – Chapter Review Questions

Apart from a solid understanding of key concepts, being able to think quickly under time pressure is a skill that will help you ace your certification exam. That is why working on these skills early on in your learning journey is key.

Chapter review questions are designed to improve your test-taking skills progressively with each chapter you learn and review your understanding of key concepts in the chapter at the same time. You’ll find these at the end of each chapter.

How To Access These Resources

To learn how to access these resources, head over to the chapter titled Chapter 21, Accessing the Online Practice Resources.

To open the Chapter Review Questions for this chapter, perform the following steps:

1. Click the link – https://packt.link/SCSC02E2_CH07

   Alternatively, you can scan the following QR code (Figure 7.17):

Figure 7.17: QR code that opens Chapter Review Questions...