Web Proxy Logs Analysis

The web proxy is one of the most critical network security controls deployed in the network as it is necessary to manage and control communications between internal users and web servers. To do so, the web proxy gets visibility of web communication aspects such as the accessed domain and web resources, web category, and user agent, which allows the proxy to generate useful logs to allow cybersecurity professionals to detect and investigate several threats, such as access to malicious websites and C&C communications. As a SOC analyst, you should be aware and take advantage of the logs provided by the web proxy and be able to analyze them to investigate cyber incidents.

The objective of this chapter is to learn the value of the web proxy logs and the provided information in the proxy logs and understand the valuable fields of the proxy logs, such as the log timestamp, source IP, source port, destination IP, destination port, response status code, username...

Understanding the value of proxy logs

A web proxy is a device that talks to external websites and domains on behalf of the clients. It is just like when you request your mother to ask your father for money on your behalf. In this case, your mother acts as a proxy between you and your father, allowing her to know crucial information such as the requested amount, the purpose behind the request, the timing of the request, and your father’s response. Similarly, within a digital network, when dealing with malicious activities and command-and-control communications, imagine the valuable information that the proxy has and how we can use it for detection, hunting, and investigation. See Figure 10.1:

Figure 10.1 – Proxy definition diagram

From a technical standpoint, when your clients access the web via a web proxy, the process should be as follows: when a client initiates a web request to access the Domain.org web server, instead of directly sending...

The significance of proxy log investigation

Before diving into the anatomy of proxy logs, let us first provide an overview of the diverse range of cyber threats that can be effectively detected and investigated by leveraging proxy logs:

• Command-and-control communication: By examining proxy logs, SOC analysts can identify anomalous patterns of traffic, such as multiple connections to known malicious domains, unusual communication patterns, or unusual communication protocols. In the next chapter, we will deep dive into the aspects of C&C communication.
• Data exfiltration: Proxy logs play a crucial role in detecting data exfiltration attempts. Unusual outbound traffic patterns, such as large volumes of data transferred to suspicious external domains or unexpected file uploads, can indicate unauthorized data exfiltration.
• Malicious file downloads: Proxy logs can reveal instances where users within the network have downloaded suspicious files from external sources. By...

The anatomy of proxy logs

The proxy generates comprehensive logs that contain a wealth of valuable information. By gaining a deep understanding of these logs and the insights they provide, you will be able to effectively investigate and detect several malicious activities.

To provide a comprehensive explanation, we will utilize a log sample generated from Blue Coat, a commercial web proxy. While it is important to note that various proxies may differ slightly in log structure and details, the concepts and insights discussed here are applicable across different proxy solutions:

Bluecoat|src=10.10.10.10|srcport=50639|dst=65.254.244.180|dstport=443|username=mostafa.yahia|devicetime=[25/10/2018:14:16:16 GMT]|s-action =TCP_Denied|sc-status=407|cs-method=GET|time-taken=256|sc-bytes=1307|cs-bytes=953|cs-uri-scheme=https|cs-host=Domain.org|cs-uri-path=/login.htm | cs-uri-extension=htm|cs-auth-group=Domain name\Admins|rs(Content-Type)=application/json;%20charset=utf-8|cs(User-Agent)=Mozilla...

Summary

In this chapter, we have discussed the value of web proxy logs, the information provided in the proxy logs, and the valuable fields of the proxy logs, such as log timestamp, source IP, source port, destination IP, destination port, response status code, username, user agent, device action, sent bytes, received bytes, referrer URL, accessed domain and URL, HTTP method, and website category.

In the next chapter, we will use the proxy logs explained in this chapter to investigate aspects of malicious web communication (C&C communication).