Penetration testing, also known as pen testing, is an authorized attack on a computer system that is done to evaluate the security of the system/network. The test is performed to identify vulnerabilities and the risks they pose. A typical penetration test is a five-stage process that identifies the target systems, their vulnerabilities, and the exploitability of each vulnerability. The goal is to find as many vulnerabilities as possible and report back in a universally acceptable format for the client to understand. Let's look at the different types of penetration testing in the next section.

Depending upon the client's requirement, penetration tests can be categorized into three types:

• White box
• Black box
• Gray box

We will discuss each of these in the following sections.

A white box penetration test, or a glass box or clear box penetration test, is a type of test in which the information and details regarding the target system, network, or application are fully shared by the client, such as the login credentials of the systems, the SSH/Telnet login for the network devices, and the application source code that needs to be tested. Since the information retrieved from the client regarding their system, network, or application is highly sensitive, it is recommended that you have all the information in an encrypted format.

A black box penetration test is an attacker-simulated test in which the penetration tester will act as a threat actor with no internal information regarding the targeted systems, networks, or applications. This type of testing really focuses on the first phase of penetration testing—reconnaissance. The more a pen tester can gain information about a target organization, the better the results will be. In this type of test, the pen tester is not provided with any architectural diagrams, layouts of the network, or any source code files.

A gray box penetration test is the halfway point between the white box and black box test. In a typical gray box test, the pen tester is provided with some knowledge of the applications, systems, or networks. Because of its nature, this type of test is quite efficient and more focused on an organization that has a deadline in place. Using the information provided by the client, the pen tester can focus on the systems with greater risks and save a lot of time performing their own recon.

Now that we have a clear understanding of the types of pen tests that can be done, let's look at the stages of a penetration test.

To have a better understanding of penetration testing, let's go through the stages of the process:

• Stage 1: Reconnaissance
• Stage 2: Enumeration
• Stage 3: Vulnerability assessment and analysis
• Stage 4: Exploitation (includes the post-exploitation period)
• Stage 5: Reporting

This can be seen in the following diagram:

Each and every stage has its own set of tools and techniques that can be used to perform the testing efficiently.

Reconnaissance is the very first stage of performing a penetration test. In this stage, a pen tester will try to identify the system or application in question and find as much information as they can about it. This is the most crucial stage of testing as this step defines the attack surface. In white box testing, the recon may not be important because all the information regarding the in-scope target is already provided by the client.

The black box test heavily relies on this stage as no information is given to the tester. In the context of a web application penetration test, we will be focusing on identifying the technology used by the web application, the domain/subdomain information, the HTTP protocol recon and enumeration, and any other details that could help us increase our efficiency. The scope for the target and the goal are generally defined at this stage.

The following is the list of tools that can be used to perform recon on a web application:

• Identifying applications running on a nonstandard port (user-defined custom ports): Amap, Nmap, and so on
• Identifying the DNS and subdomains: dnsenum, dnsmap, dnswalk, dnsrecon, dnstracer, Fierce, dnscan, Sublist3r, and so on
• Identifying technological platforms: BlindElephant, Wappalyzer, WhatWeb, and so on
• Identifying content management systems: WPScan, Joomscan, CMScan, Drupscan, and so on

Now, let's look at enumeration.

In the enumeration stage, each and every application, system, or network identified in the previous stage (recon) will be scanned for different attack surfaces—for example, files and directory enumeration in the case of a web application, and ports and services in the case of a network device. This stage will help the tester to identify the attack vectors. An attack vector is a path or method for the attacker to gain access or penetrate the target system; in this case, the pen tester. The most common attack vectors used are phishing emails, malware, and unpatched vulnerabilities.

A pen tester can perform file and directory enumeration, HTTP method enumerations, host enumeration, and a few other enumeration methods to find an insertion point where vulnerabilities might exist. In a white box test, this stage doesn't really play an important role as all the information and details are already given to the tester, but it doesn't mean that you should not go through with this stage. It's always a good practice to perform enumeration and scanning, even when all the details are provided. This will help the tester to find obsolete attack paths that are not supported by the application but may help the tester to penetrate the network.

This stage is very crucial for the black box and gray box test as all the information that was retrieved by performing reconnaissance on the target system or application is identified by the pen tester. Enumeration could become a tedious process if done manually, so there are publicly available tools and some Metasploit modules that can be used to enumerate applications quickly.

The following is a list of tools that can be used to perform enumeration on a web application:

• Files and directory enumeration: Dirsearch, dirb, dirbuster, Metasploit Framework, BurpSuite, gobuster, and so on
• HTTP protocol supported methods enumeration: Nmap, BurpSuite, Metasploit Framework, wfuzz, and so on
• Testing for rate limiting: BurpSuite, ffuf, wfuzz, and so on

Let's now look at vulnerability assessment.

Once we have identified an attack vector, we need to perform vulnerability scanning, which occurs in this stage of penetration testing. A vulnerability assessment is done on the web application to identify vulnerabilities on a web page, directory, HTTP protocol method, HTTP headers, and so on. The Scanning can be done using publicly available tools or paid-for licensed tools. All types of testing—white box, black box, and gray box— rely heavily on this stage.

Once a vulnerability scan has been done, we need to assess and analyze each vulnerability that is found and then filter out the false positives. Filtering out the false positives helps the pen tester to work on the vulnerabilities that actually exist and not the ones that were found because of time delay or the scanner's error. All the vulnerability filtration happens at this stage.

The following is the list of tools that can be used to perform vulnerability assessment and scanning on a web application:

• System and network vulnerability assessment: Nessus, OpenVAS, and so on
• Web application vulnerability assessment: Nikto, Acunetix, BurpSuite, Nessus, and so on

The exploitation stage is the second most crucial stage after the reconnaissance stage. This stage proves whether a certain vulnerability found in the previous stage is exploitable. A pen tester can always identify the success of penetration testing projects if they can exploit the vulnerabilities that are found. Exploitation can be done automatically using certain tools, such as Metasploit Framework and Canvas. This is because we don't know how a certain web application or system will behave when we use our payloads.

Generally, in all types of tests, we need to confirm from the client whether we are authorized to perform memory-based exploitation, such as exploiting buffer/heap overflows and running memory corruption exploits. The advantage of doing this is that we can have access to the target system by running a specific exploit (this only works if the target system is vulnerable to this specific exploit). The issue with using such exploits is that the system/server/web application may crash, which could cause a business continuity issue.

Once we have exploited a system or web application, we can either stop at that or we can perform post-exploitation work (if authorized by the client) to move inside the network (pivoting) and locate business-critical servers.

Please make sure that all the payloads, web shells, files, and scripts are uploaded to the target system for exploitation so that they can be cleaned up after taking proper proof-of-concept (PoC) screenshots. This should be done at all times; otherwise, a genuine attacker can find the web shells and easily use them to attack the organization.

The reporting stage is the final stage of the penetration testing process and involves reporting each and every vulnerability found on the target (in-scope). The reported vulnerabilities will be listed according to the severity level defined by the Common Vulnerability Scoring System (CVSS), which is a free and open standard that is used to assess the vulnerabilities.

As pen testers, we need to understand how important this stage really is for the client. All the work that has been done by the testers on the client system should be reported in a structured format. The report should include a short introduction to the test, the scope of work, the rules of engagement, a short and crisp summary, the vulnerabilities found, and the proof of concept for each vulnerability, with some recommendations and patching techniques from the reference links.

There are some publicly available tools, such as Serpico, Magic Tree, BurpSuite, and Acunetix that can be used to ease the process of reporting. As this is an important stage of pen testing, all the details that were found during the test should be included in the report.

We can provide two different kinds of report: an executive report for management and a technical report for the technical team in place. This could help both the management and the technical team of an organization to understand and fix the vulnerabilities found by the penetration testers.

Now that we are familiar with the standards, let's now cover the important terminology that we will be using a lot in the upcoming chapters:

• Vulnerability: A weakness in a system that may allow an attacker to gain unauthorized access to it.
• Spoofing: A situation where an individual or program successfully masks data as something else in order to obtain an unlawful advantage.
• Exploit: A piece of code, a program, a method, or a sequence of commands that takes advantage of a vulnerability to gain unauthorized access to a system/application.
• Payload: The actual code that is executed on the system after/during exploitation to perform the desired task.
• Risk: Anything that can affect the confidentiality, integrity, and availability of data. Unpatched software, misconfigured servers, unsafe internet surfing habits, and so on all contribute to risk.
• Threat: Anything that may have the potential to cause serious harm to a computer system, network, or application.
• Black box: A method of testing during which the tester has no information about the internal structure or functioning of a system.
• White box: A method of testing during which the tester has complete knowledge of the internal structure and functioning of a system.

• Bug bounty: A bug bounty program is a deal that is offered by many websites and developers that allows individuals to be honored and rewarded for reporting bugs, particularly those linked to exploits and vulnerabilities.
• SAST: Static application security testing (SAST) is a form of security testing that relies on the inspection of an application's source code.
• DAST: Dynamic application security testing (DAST) is a technique that is used to detect security vulnerabilities in an application in its running state.
• Fuzzing: An automated testing technique in which invalid, unexpected, or random data is provided as input to an application.

Now that we are aware of this important terminology, let's go ahead and learn about testing methodologies.

As we all know, there are no official penetration testing standards defined; however, our security community has introduced a few standards for all security personnel to follow. Some of the commonly known standards are the Open Source Security Testing Methodology Manual (OSSTMM), the Penetration Testing Execution Standard (PTES), and the Information Systems Security Assessment Framework (ISSAF). Most of them follow the same methodology, but their phases have been named differently. We will take a look at each of them in the following sections and cover PTES in detail.

The definition of the OSSTMM is mentioned on their official website, at https://www.isecom.org/OSSTMM.3.pdf:

Using the OSSTMM, an audit will provide a precise estimation of security at an operational level that clears out assumptions and unreliable evidence. It is used for thorough security testing and is designed to be consistent and repeatable. As an open source project, it is open to contributions from all security testers, encouraging increasingly accurate, actionable, and productive security tests.

OSSTMM includes the following key sections:

• Operational security metrics
• Trust analysis
• Human security testing
• Physical security testing
• Wireless security testing
• Telecommunications security testing
• Data network security testing
• Compliance regulations
• Reporting with the Security Test Audit Report (STAR)

This part of the OSSTMM section deals with what needs to be protected and how much the attack surface is exposed. This can be measured by creating an RAV (an unbiased factual description of the attack surface).

In operational security, trust is measured as the interactions between targets within the scope that can be exploited by any person with malicious intent. To quantify trust, we need to understand and perform analysis to make more rational and logical decisions.

Human Security (HUMSEC) is a subsection of Physical Security (PHYSSEC) and incorporates Psychological Operations (PSYOPS). Testing this aspect of security requires communication with individuals who have physical access to the protected assets—for example, a gatekeeper.

Physical Security (PHYSSEC) refers to material security inside the physical domain. Testing this channel requires noncommunicative interaction with barriers and humans (gatekeepers) placed within the assets.

Spectrum Security (SPECSEC) is the security classification that includes Electronics Security (ELSEC), Signals Security (SIGSEC), and Emanations Security (EMSEC). Testing this channel requires the analyst to be within the vicinity of the target.

Telecommunications Security is a subset of ELSEC, which describes the organization's telecommunication over wires. Testing this channel covers the interaction between the analyst and the targets.

Tests regarding the Data Network Security (Communications Security [COMSEC]) aspect of security requires interaction with the individuals who have access to the operational data that is used to control access to the property.

The kind of compliance required depends on the locale and currently ruling government, industry and business types, and supporting legislation. In a nutshell, compliance is a set of general policies that are defined by the legislation or the industry, and these policies are compulsory.

The purpose of a Security Test Audit Report (STAR) is to serve as an executive summary, stating the attack surface of the targets tested within a particular scope.

OSSTMM divides the testing types into six broad categories based on the amount of information known to the tester:

• Blind: In this test, the analyst has no knowledge of the target, but the target knows about the audit and has all the details of the analyst. This can be considered a test of the analyst's knowledge.
• Double-Blind: In this test, the analyst has no knowledge of the target, its defenses, assets, and so on. The target is also not notified of the audit. This test is used to check the knowledge and skills of the analyst as well as the preparedness of the target against unknown threats. This is also known as a black box test.
• Gray Box: In this test, the analyst has limited knowledge of the defenses of the target, but has complete knowledge of the assets and workings of the target. The target, in this case, is fully prepared for the audit and knows its full details. This test is also referred to as a Vulnerability Test.
• Double Gray Box: This is also known as the white box test. The target has advance knowledge of the scope and timeframe but has no knowledge of the payloads and test vectors.
• Tandem: This is also referred to as an in-house audit or crystal ball test. In this test, both the target and the analyst know the full details of the audit, but this test does not check the preparedness of the target against unknown variables or vectors.
• Reversal: In this test, an attacker engages with full knowledge of its target's processes and operational security, but the target doesn't know anything about when or how the audit will happen. This is also referred to as a red team exercise.

Here are these types represented in a graph:

Now that we have read through the different OSSTMM test types, let's look at ISSAF.

The ISSAF is not very active, but the guide they have provided is quite comprehensive. It aims to evaluate information security policy and an organization's compliance with IT industry standards, laws, and regulatory requirements. The current version of ISSAF is 0.2.

It covers the following stages:

• Project management
• Guidelines and best practices—pre-assessment, assessment, and post-assessment
• Assessment methodology
• Review of information security policy and security organization
• Evaluation of risk assessment methodology
• Technical control assessment
• Technical control assessment—methodology
• Password security

• Password cracking strategies
• Unix /Linux system security assessment
• Windows system security assessment
• Novell netware security assessment
• Database security assessment
• Wireless security assessment
• Switch security assessment
• Router security assessment
• Firewall security assessment
• Intrusion detection system security assessment
• VPN security assessment
• Anti-virus system security assessment and management strategy
• Web application security assessment
• Storage area network (SAN) security
• Internet user security
• As 400 security
• Source code auditing
• Binary auditing
• Social engineering
• Physical security assessment
• Incident analysis
• Review of logging/monitoring and auditing processes
• Business continuity planning and disaster recovery
• Security awareness and training
• Outsourcing security concerns
• Knowledge base
• Legal aspects of security assessment projects
• Non-disclosure agreement (NDA)
• Security assessment contract
• Request for Proposal Template
• Desktop security checklist—windows
• Linux security checklist
• Solaris operating system security checklist
• Default ports—firewall
• Default ports—IDS/IPS

This standard is the most widely used standard and covers almost everything related to the pen test.

PTES is divided into seven phases:

• Pre-engagement interactions
• Intelligence gathering
• Threat modeling
• Vulnerability analysis
• Exploitation
• Post exploitation
• Reporting

Let’s take a brief look at what each of these phases involves.

Pre-engagement interactions are carried out before an activity kicks off, such as defining the scope of the activity, which usually involves mapping the network IPs, web applications, wireless networks, and so on.

Once the scoping is done, lines of communication are established across both the vendors and the incident reporting process is finalized. These interactions also include status updates, calls, legal processes, and the start and end date of the project.

Intelligence gathering is a process that is used to gather as much information as possible on the target. This is the most critical part of pen testing as the more information we have, the more attack vectors we can use to perform the activity. In case of a white box activity, all this information is already provided to the testing team.

Threat modeling is a process by which potential threats can be identified and enumerated and mitigations can be prioritized. Threat modeling depends on the amount and quality of information gathered; with this information, the activity can be broken down into stages and then performed using automated tools and logical attacks.

The following is a mind map of a threat model:

Let's now have a look at vulnerability analysis.

Vulnerability analysis is a process of discovering flaws that can be used by an attacker. These flaws can be anything ranging from open ports and service misconfigurations to an SQL injection. There are lots of tools available that can help in performing a vulnerability analysis—for example, Nmap, Acunetix, and Burp Suite. New tools are currently being released every few weeks.

Exploitation is the process of gaining access to the system by evading the protection mechanism based on the vulnerability assessment. Exploits can be public or zero-day.

Post-exploitation is the stage where the goal is to determine the criticality of the compromise and then maintain access for future use. This phase must always follow the rules of engagement that protect the client and protect ourselves (covering the tracks as per the requirements of the activity).

Reporting is one of the most important phases, as patching all the issues wholly depends on the details presented in your report. The report must contain three key elements:

• The criticality of the bug
• The steps needed to reproduce the bug
• Patch suggestions

In summary, the pen test life cycle phases can be presented in the following way:

In the next section, we will talk about the Common Weakness Enumeration (CWE) and the two top CWEs.

In this section, we will talk about the Common Weakness Enumeration (CWE). The CWE is a universal online dictionary of weaknesses that have been found in computer software. In this section, we will cover two well-known CWEs—the OWASP Top 10 and the SANS Top 25.

Open Web Application Security Project (OWASP) is an organization that provides computer and internet applications with impartial, realistic, and cost-effective information.

The current list for 2020 contains the following bugs:

• Injection
• Broken authentication
• Sensitive data exposure
• XML external entities (XXE)

• Broken access control
• Security misconfigurations
• Cross-site scripting (XSS)
• Insecure deserialization
• Using components with known vulnerabilities
• Insufficient logging and monitoring

The SANS Top 25 list is a collaboration between the SANS Institute, MITRE, and many top software security experts in the US and Europe. It consists of the following vulnerabilities:

• Improper neutralization of special elements used in a SQL command ('SQL injection')
• Improper neutralization of special elements used in an OS command ('OS command injection')
• Buffer copy without checking the size of the input ('classic buffer overflow')
• Improper neutralization of the input during web page generation ('cross-site scripting')
• Missing authentication for a critical function
• Missing authorization
• Use of hardcoded credentials
• Missing encryption of sensitive data
• Unrestricted upload of a file of a dangerous type
• Reliance on untrusted inputs in a security decision
• Execution with unnecessary privileges
• Cross-site request forgery (CSRF)
• Improper limitation of a pathname to a restricted directory ('path traversal')
• The downloading of code without an integrity check
• Incorrect authorization
• Inclusion of a functionality from an untrusted control sphere
• Incorrect permission assignment for a critical resource
• Use of a potentially dangerous function
• Use of a broken or risky cryptographic algorithm
• Incorrect calculation of buffer size

• Improper restriction of excessive authentication attempts
• URL redirection to an untrusted site ('open redirect')
• Uncontrolled format string
• Integer overflow or wraparound
• Use of a one-way hash without a salt

We will cover some of these vulnerabilities in detail in later chapters of this book.

In this chapter, we started with the introduction to penetration testing and its types and stages. We covered the pen testing methodologies and life cycle and we looked at some important terminology. Then, we looked at the OWASP Top 10 and SANS Top 25.

In the next chapter, we will learn about the essentials of Metasploit including the Metasploit framework, installation, and setup.

1. Is there a database that maintains the Common Weakness Enumeration (CWE) list?

2. Where can I find the OWASP Top 10 and SANS Top 25 lists?

3. Are the tools required to perform a penetration test free?

4. How do the OSSTMM- and PTES-based penetration tests differ?

• The Institute for Security and Open Methodologies (ISECOM): http://www.isecom.org/
• The pen test standard website: http://www.pentest-standard.org/index.php/Main_Page