Security Logging and Monitoring

In simple terms, a log is a record of an event that has occurred within the systems and networks of an organization.

When we speak about security, the logs can be generated by various sources such as antivirus, firewalls, intrusion prevention systems, and operating system.

Nowadays, organizations have hundreds of servers and logging into each of them and checking security-related events is not a feasible solution.

This is one of the reasons why bringing all logs to a single place where it can be stored and analyzed is very important. Centralized logging and archiving is also a regulatory requirement. This is the reason why organizations are moving toward a concrete log monitoring solution.

This is further illustrated in the following diagram:

In the preceding diagram, we see that logs from various devices such as firewalls, switches, and routers...

Continuous security and monitoring

In June 2017, one of the mid-sized organizations was breached and the entire database was downloaded by the hackers. The irony was that the organization didn't even know anything until the hacker was selling the database details on the dark web after a few months.

Even though they had implemented decent security controls, the thing that was lacking was a continuous security monitoring. The lack of continuous security monitoring is one of the most common things that you might find in a startup or mid-sized organization.

This is one of the reasons why continuous security monitoring needs to become an integral part of the modern security architectures.

Continuous security monitoring gives us the ability to trace what exactly is happening within the environment in a timely manner, as timely detection is important to implement prevention.

Having...

Choosing the right log monitoring tool

This is a very important decision that needs to be taken by the organization. There are both commercial offerings as well as open source offerings that are available today but the amount of efforts that need to be taken in each of them varies a lot.

I have seen many commercial offerings such as Splunk and ArcSight being used in large enterprises, including national level banks. On the contrary, there are also open source offerings, such as ELK Stack, that are gaining popularity especially after Filebeat got introduced.

At a personal level, I really like Splunk but it gets very expensive when you have a lot of data being generated. This is one of the reasons why many startups or mid-sized organizations use commercial offering along with open source offerings such as ELK Stack.

Having said that, we need to understand that if you decide to go...

Security Incident and Event Management

Security Incident and Event Management (SIEM) enhances the power of a traditional log monitoring tool with the help of co-relation and alerting-based solution.

Co-relation is one powerful feature that makes SIEM a distinguished player.

Let's understand the co-relation part with an example. The user's ID card has been swiped in at the office; however, his ID card was swiped at the datacenter provider as well without having swiped out at the office. This seems suspicious. SIEM will co-relate the two events and can determine that the user cannot be present at both the locations simultaneously and will alert the SOC immediately.

There was a possible port scan from a particular IP address and then there was a possible login attempt to an SSH service running on an ephemeral port. These two events are co-related and need to be alerted...

Log monitoring is reactive in nature

If we look into an overall log management activity, it comprises three phases:

• Generation of logs from sources
• Delivery of logs to central solution
• Co-relation and alerting based on rules

In each of these phases, there is some kind of time involved and thus at the final stage when the SOC gets an alert of some suspicious activity, some time would have already passed before the activity is actually being performed:

There are certain phases involved, which are explained as follows:

• N: This is the time when the user makes a certain request to the system
• N+1: This is the time when the system will log the request in the appropriate log file
• N+X: This is the time when the logs from the system will go to the central log monitoring solution
• N+X+Y: This is the time taken by the log monitoring solution to co-relate the logs and display an alert

...

Summary

In this chapter, we had a overview about log monitoring challenges, best practices, and certain tools that will help us on the way. Since we have specifically discussed the advantages of auditd in Linux in Chapter 4, Server Hardening, to determine the system events, this chapter was more focused on the process and tools part. In the next chapter, we will discuss more related to the incident response regarding what happens when a critical incident is detected through your monitoring solutions.