Parsing Logs and Events with AWS Native Tools

The previous chapter showed you how the CloudWatch service can help collect and store logs from a myriad of services in AWS. You are now ready to turn your attention to the most cost-effective ways to retain those log files for long-term storage, along with the methods to pull out the necessary data from them.

One of the critical duties of a security professional is to assimilate all the information coming in from different sources and distinguish the relevant bits of information from that which is just noise. Services and applications in any environment (not just the cloud) constantly produce logs. Knowing which services and techniques can gather, collect, and then help you quickly sift through and analyze these logs is an essential skill for real-life situations as well as for the AWS Security competency exam.

Several services can help you with this task. This chapter will cover such services, including storing logs on the S3 storage...

Technical Requirements

You will require access to an active AWS account, the AWS Management Console, the CLI, and a text editor for this chapter.

Log Storage Options and Their Cost Implications

As you think about storing all the logs generated in your account, there are a few different factors that you should consider as you come up with long-term solutions for log storage for your organization:

• Building a storage solution that is both secure and resilient: Your logs should be stored in a secure manner that includes, at a minimum, default encryption on that log storage. Furthermore, the space you create for the log storage should be able to store all of your logs in real time without delays in processing and storing.
• Central storage for the log files: You need a location to direct any internal or external auditors should they need access to the log files generated for your account. This is also true for any configuration changes that have occurred in the different accounts you manage, as the logs need to be stored in a centralized location both for auditors and in case of access for incident response.
• Establishing...

Moving Logs from CloudWatch Logs

Although CloudWatch Logs provides a sustainable logging solution right away, storage can be a little more costly than other solutions for logs that are not being utilized and just need to be kept for compliance purposes. Additionally, the service does not have the ability to search across multiple log groups at the same time. If you need to either cut costs for your log storage or search across multiple streams of logs for trends and anomalies, you will need to move those logs to another storage system.

It all begins with the use of CloudWatch subscription filters. As you will see in the following exercise, subscription filters allow you to push all or a subset of the logs that are coming into the CloudWatch Logs group to a Kinesis Data Firehose stream or a Lambda function, or even export the data to an S3 bucket.

Figure 9.7: Moving log files from CloudWatch Logs to S3

The next section will dive deeper into subscription...

Using Amazon Kinesis to Process Logs

Amazon Kinesis Firehose is a fully managed service that can scale automatically with the volume of the data throughput that you are sending to it. It can scale automatically if large quantities of data suddenly appear and then scale back down again. Data can be encrypted and sent to its final destination using the Firehose service.

The following are some key facts to understand about Amazon Kinesis Firehose:

• Kinesis Firehose is serverless.
• Kinesis Firehose is elastic and can scale automatically based on data volume.
• Kinesis Firehose can transform data with the help of AWS Lambda.
• Kinesis Firehose can deliver data to AWS OpenSearch Service, Amazon Redshift, Splunk, or Amazon S3.
• Kinesis Firehose can handle errors and the buffering of data but does not have the capability to replay data.

Kinesis Firehose can also transform the data that passes through it before it is delivered to its final destination. An example...

Running Queries with Amazon Athena

With so much valuable data within CloudTrail and other logs, finding effective ways to query the data for specific entries is always a top priority. The Amazon Athena service makes running ad hoc queries on extensive datasets much more straightforward. When discussing storing logs on the S3 service earlier in the chapter, one of the feature’s shortcomings was the ability to query the logs. This gap in capabilities is filled by using the Amazon Athena service.

Amazon Athena is a serverless service that allows you to quickly analyze data stored within Amazon S3, such as your CloudTrail logs. Athena does this by using an interactive query service and letting you write your queries using standard Structured Query Language (SQL). As a result, it is an efficient service to help you scan massive datasets.

The following are some key facts to understand about Amazon Athena:

• Athena separates storage from compute by utilizing Amazon S3 for...

Storing and Searching Logs in Amazon OpenSearch Service

As logs and other pieces of data you are responsible for securing are generated in your environment, be sure to consider your security goals for the data itself and your organization as a whole. Ask yourself whether these goals include the following:

• Protecting confidential business data
• Maintaining business access controls
• Having the ability to audit user actions
• Possessing the ability to integrate with SAML identity providers
• Keeping your systems and data compliant with a myriad of compliance frameworks such as HIPAA, SOC, PCI, and others

If the answer is yes to one or more of these items, then provisioning an Amazon OpenSearch Service cluster could help meet your needs.

Amazon OpenSearch Service is a search and analytics engine developed to be compatible with Elasticsearch, a popular search engine based on Elasticsearch provided by the Elastic company. In addition to the ability to store and...

Summary

In this chapter, you learned how to review and analyze many log files using native tools found in AWS.

You also had a brief overview of how costs can affect your decisions when choosing the correct storage solution for your long-term log storage for your account and your organization. This overview was presented through the different tiers of storage available in the S3 service and their durability and reliability ratings.

Next, you explored how to move logs out of CloudWatch Logs using subscription filters. You learned that combining CloudWatch subscription filters with the Kinesis Data Firehose service allows you to take incoming logs to CloudWatch Logs and push them into an S3 bucket where they could be stored at lower costs and then be queried by the Amazon Athena service.

You also examined how you could use Kinesis Data Firehose to stream the logs to Amazon OpenSearch, the fast search engine with custom visualizations based on Elasticsearch.

The next chapter...

Further Reading

For additional information on the AWS shared responsibility model and the underlying foundation of AWS security, please look at the following resources:

• AWS Pricing Calculator: https://packt.link/o2kod
• Managing your storage lifecycle (in S3): https://packt.link/2TQrC
• Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis: https://packt.link/Ij7my
• Amazon OpenSearch FAQs: https://packt.link/Wz8Cb
• Amazon Athena FAQs: https://packt.link/CSKmD

Exam Readiness Drill – Chapter Review Questions

Apart from a solid understanding of key concepts, being able to think quickly under time pressure is a skill that will help you ace your certification exam. That is why working on these skills early on in your learning journey is key.

Chapter review questions are designed to improve your test-taking skills progressively with each chapter you learn and review your understanding of key concepts in the chapter at the same time. You’ll find these at the end of each chapter.

How To Access These Resources

To learn how to access these resources, head over to the chapter titled Chapter 21, Accessing the Online Practice Resources.

To open the Chapter Review Questions for this chapter, perform the following steps:

1. Click the link – https://packt.link/SCSC02E2_CH09

   Alternatively, you can scan the following QR code (Figure 9.17):

Figure 9.17: QR code that opens Chapter Review Questions...