Detection Validation

Cyber security defenses are designed to protect a company’s information assets. This amalgamation of trained personnel, specialized technology, and underlying processes can be designed based on security best practices, threat modeling outputs, technical knowledge, available threat intelligence, and expert judgment. Once implemented, we need assurance that controls work as expected, under realistic conditions. Cyber security validation aims to create these conditions, that is, the techniques, tactics, and procedures used by threat actors, to measure the effectiveness of defensive control.

Cyber security validation has the goal of producing tangible measurements of how well a security program is performing. For detection engineering, well-executed validations give us the opportunity to find weaknesses or blind spots in our detection environment and remediate them before they can create an advantage for threat actors.

The process and techniques used to...

Technical requirements

The exercises in this chapter require the following:

• The Elastic Stack (as configured in Chapter 3)
• A Windows VM:
  • Minimum RAM: 4 GB
  • Minimum disk space: 80 GB
• A Linux VM:
  • Minimum RAM: 8 GB
  • Minimum disk space: 10 GB

An important note on the tools in this chapter

Most of the tools in this chapter are adversary emulation tools, meaning that they are designed to perform activities similar to those of a real attacker. As such, be mindful of the systems and networks on which you are running the tools as if not used properly, you could impact systems accidentally. They should also be used after a proper review of the documentation and at the user’s own risk.

Understanding the validation process

The execution of cyber security validation is very similar to typical adversary simulation exercises. The emphasis, however, is on producing data that can be compared against a set of performance criteria defined for each defensive control. In broad terms, validation can be executed in three phases:

1. Planning: This is easily the most important phase. During this phase, the objectives of the validation exercise are defined, along with the scope, timelines, and stakeholders. The specific defensive capabilities targeted for validation and the criteria for determining their effectiveness are rigidly defined during this phase. Each validation needs to be mapped to a specific defensive control or controls, expected outcomes, and criteria for measuring the performance of the control(s). It is important at this time to also understand the possible limitations of each validation. For example, an organization may want to test T1048: Exfiltration over...

Understanding purple team exercises

Security functions can be broadly organized into two categories: the blue team, which focuses on defending an organization against cyber security threats, and the red team, which has the goal of emulating real-world adversaries. When the red and blue teams work together, collaboratively, to emulate an adversary, execute tactical defensive activity (where relevant), observe the performance of security controls, and execute responses in real time, this is referred to as a purple team exercise. While developed detections do get tested during a purple team exercise, the central focus of the exercise is not just the detection environment but rather the interactions between the red and blue teams. The exercises aim to help the blue team develop and improve response techniques while simultaneously helping the red team develop adversarial techniques.

Both teams work together to plan a simulated cyber-attack, comprising several tactics, within a predefined...

Simulating adversary activity

For our detection lab, we may not have a red team readily available, but we still need to track how well our detections respond to realistic threat actor techniques. Fortunately, there are some free and publicly-available breach and attack simulation (BAS) resources we can use to emulate adversary behavior. We cover some noteworthy, freely available options in this section.

An important note on impairing security tools

Some validation tools and techniques can get blocked by different security controls, which is normally a good thing. However, this might prevent the validation exercise from being run as required. A preventative control on an endpoint can in some cases limit our ability to validate detective controls.

For example, consider the scenario where we need to validate detections for the creation of the log file associated with executing the mimikatz misc::memssp module. If we run mimikatz, but it immediately gets blocked and removed by...

Using validation results

After performing validations, we will walk away with some understanding of coverage. In the simplest form, validation results identify whether a detection is triggered for a given behavior. If we run a command to set persistence via a Registry Run key during validation, and we expect a detection to be triggered by that activity, we can record the result as either failed or successful by reviewing our triggered detections. Validation results, however, can also operate on more of a scale. For example, maybe a detection triggered but it was not the specific rule we expected. Or maybe we have variations of Registry Run key persistence and some were detected but others weren’t, in which case we might have partial coverage. Therefore, validation results are not always black-and-white, but they will provide some level of guidance as to what happens when a chosen test is executed, which can be leveraged to improve our detection engineering program.

With the...

Summary

In this chapter, we provided an overview of validation as it relates to assessing the maturity of a detection engineering program. We then introduced a series of open source tools that can be leveraged by organizations without purple teams to perform validation tests easily and effectively. The exercises in the chapter leveraged some of these tools to simulate adversary activity in order to validate detections. Finally, we concluded the chapter by explaining how the results of validation tests can be leveraged to improve your detection posture.

In the next chapter, we will take a look at the topic of threat intelligence. We’ll discuss the different types of threat intelligence and the roles they can play in detection engineering.

Further reading

While we focused on Atomic Red Team and CALDERA during our discussion of breach and attack simulation (BAS) tools, there are many commercial and open source solutions out there that can be used as alternatives. Let’s highlight two popular open source options:

• Infection Monkey:

Infection Monkey is an open source platform that can be used for launching realistic attacks against a specified set of target endpoints: https://www.akamai.com/infectionmonkey

• Network Flight Simulator (flightsim):

AlphaSOC’s Network Flight Simulator (flightsim), focuses instead on generating malicious-looking network traffic, specifically for testing detections built for network telemetry: https://github.com/alphasoc/flightsim