Log Analysis

In Chapter 14, Investigating an Incident, you learned about the investigation process, and some techniques for finding the right information while investigating an issue. However, to investigate a security issue, it is often necessary to review multiple logs from different vendors and different devices. Although each vendor might have some custom fields in the log, the reality is that once you learn how to read logs, it becomes easier to switch vendors and just focus on deltas for that vendor. While there are many tools that will automate log aggregation, such as a SIEM solution, there will be scenarios in which you need to manually analyze a log in order to figure out the root cause.

In this chapter, we are going to cover the following topics:

• Data correlation
• Operating system logs
• Firewall logs
• Web server logs
• Amazon Web Services (AWS) logs
• Azure Activity logs
• Google Cloud Platform (GCP) logs

Let’s start...

Data correlation

There is no doubt that the majority of organizations will be using some sort of SIEM solution to concentrate all of their logs in one single location, and using a custom query language to search throughout the logs. While this is the current reality, as a security professional, you still need to know how to navigate through different events, logs, and artifacts to perform deeper investigations. Many times, the data obtained from the SIEM will be useful in identifying the threat, the threat actors, and narrowing down the compromised systems but, in some circumstances, this is not enough; you need to find the root cause and eradicate the threat.

For this reason, every time that you perform data analysis, it is important to think about how the pieces of the puzzle will be working together.

The following diagram shows an example of this data correlation approach to review logs:

Figure 17.1: Data correlation approach while reviewing logs

Let’...

Operating system logs

The types of logs available in an operating system may vary; in this book, we will focus on core logs that are relevant from a security perspective. We will use Windows and Linux operating systems to demonstrate that.

Windows logs

In a Windows operating system, the most relevant security-related logs are accessible via Event Viewer. In Chapter 14, Investigating an Incident, we spoke about the most common events that should be reviewed during an investigation. While the events can be easily located in Event Viewer, you can also obtain the individual files at Windows\System32\winevt\Logs, as shown in the following screenshot:

Figure 17.2: Most relevant security-related logs

However, log analysis in an operating system is not necessarily limited to the logging information provided by the OS, especially in Windows. There are other sources of information that you could use, including prefetch files (Windows Prefetch). These files contain relevant...

Firewall logs

The firewall log format varies according to the vendor; however, there are some core fields that will be there regardless of the platform. When reviewing the firewall logs, you must focus on primarily answering the following questions:

• Who started the communication (source IP)?
• Where is the destination of that communication (destination IP)?
• What type of application is trying to reach the destination (transport protocol and port)?
• Was the connection allowed or denied by the firewall?

The following code is an example of the Check Point firewall log; in this case, we are hiding the destination IP for privacy purposes:

"Date","Time","Action","FW.Name","Direction","Source","Destination","Bytes","Rules","Protocol" "
datetime=26Nov2017","21:27:02","action=drop","fw_name=Governo","dir=inboun...

Web server logs

When reviewing web server logs, pay particular attention to the web servers that have web applications interacting with SQL databases.

On a Windows server hosting one site, the IIS web server log files are located at \WINDOWS\system32\LogFiles\W3SVC1 and they are .log files that can be opened using Notepad. You can also use Excel or Microsoft Log Parser to open this file and perform basic queries.

You can download Log Parser from https://www.microsoft.com/en-us/download/details.aspx?id=24659.

When reviewing the IIS log, pay close attention to the cs-uri-query and sc-status fields. These fields will show details about the HTTP requests that were performed. If you use Log Parser, you can perform a query against the log file to quickly identify whether the system experienced a SQL injection attack. Here is an example:

logparser.exe -i:iisw3c -o:Datagrid -rtp:100 "select date, time, c-ip, cs- uri-stem, cs-uri-query, time-taken, sc-status from C:wwwlogsW3SVCXXXexTEST...

Amazon Web Services (AWS) logs

When you have resources located on Amazon Web Services (AWS), and you need to audit the overall activity of the platform, you need to enable AWS CloudTrail. When you enable this feature, all activities that are occurring in your AWS account will be recorded in a CloudTrail event.

These events are searchable and are kept for 90 days in your AWS account. Here you have an example of a trail:

Figure 17.3: Trails shown in AWS

If you click Event history, in the left navigation, you can see the list of events that were created. The list below has interesting events, including the deletion of a volume and the creation of a new role:

Figure 17.4: Event history in AWS

This is a comprehensive list of all events that were tracked. You can click on each one of those events to obtain more detailed information about it, as shown below:

Figure 17.5: Specific event information when clicking on one of the events listed in AWS

...

Azure Activity logs

Microsoft Azure also has platform logging that enables you to visualize subscription-level events that have occurred in Azure. These events include a range of data, from Azure Resource Manager (ARM) operational data to updates on Service Health events. These logs are also stored for 90 days by default, and this log is enabled by default.

To access the Azure Activity log, go to Azure Portal; in the search box, type Activity, and once you see the Activity log icon, click on it. The result may vary, but you should see some activities similar to the sample screen that follows:

Figure 17.8: A sample of the Azure Activity log

You can expand these activities to obtain more information about each action, and you can also retrieve the raw JSON data with all the details about the activity.

Accessing Azure Activity logs from Microsoft Sentinel

If you are using Microsoft Sentinel as your SIEM platform, you can use the native Azure Activity log connector...

Google Cloud Platform Logs

Many organizations are moving towards a multi-cloud environment and Google Cloud Platform (GCP) is another big player that you need to be aware of how to monitor. GCP Cloud Audit Logs enables you to answer the following questions:

• Who did what?
• When was it done?
• Where was it done?

Using Microsoft Sentinel, you can ingest GCP Identity and Access Management (IAM) logs, which can be used to see admin activity (audit logs), which includes “admin write” operations, and Data Access audit logs, which includes “admin read” operations.

Once the connector is configured, the status will appear similar to the sample screenshot that follows:

Figure 17.11: GCP IAM connector

Once the connector is configured and ingesting data, you can perform queries using KQL. The example below is checking all GCP IAM logs and filtering the result to show only the following fields: SourceSystem, resource_labels_method_s...

Summary

In this chapter, you learned about the importance of data correlation while reviewing logs in different locations. You also read about relevant security-related logs in Windows and Linux.

Next, you learned how to read firewall logs using Check Point, NetScreen, iptables, and Windows Firewall as examples. You also learned about web server logs, using IIS and Apache as examples. You concluded this chapter by learning more about AWS CloudTrail logs, and how they can be visualized using the AWS dashboard, or Microsoft Sentinel. You also learned about Azure Activity logs and how to visualize this data using Azure Portal and Microsoft Sentinel. Lastly, you learned about GCP IAM logs and how to visualize those using Microsoft Sentinel. As you finish reading this chapter, also keep in mind that, many times, it is not about quantity, but about quality. When the subject is log analysis, this is extremely important. Make sure that you have tools that are able to intelligently ingest...

References

• iptables: https://help.ubuntu.com/community/IptablesHowTo
• Log Parser: https://logrhythm.com/blog/a-technical-analysis-of-wannacry-ransomware/
• SQL Injection Cheat Sheet: https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/

Join our community on Discord

Join our community’s Discord space for discussions with the author and other readers:

https://packt.link/SecNet