Before we take a deep dive into various aspects of the Metasploit framework, let's first lay a solid foundation of some of the absolute basics. In this chapter, we'll conceptually understand what penetration testing is all about and where the Metasploit Framework fits in exactly. We'll also browse through some of the additional tools that enhance the Metasploit Framework's capabilities. In this chapter, we will cover the following topics:

• Importance of penetration testing
• Differentiating between vulnerability assessment and penetration testing
• Need for a penetration testing framework
• A brief introduction to Metasploit
• Understanding the applicability of Metasploit throughout all phases of penetration testing
• Introduction to supporting tools that help extend Metasploit's capabilities

For more than over a decade or so, the use of technology has been rising exponentially. Almost all of the businesses are partially or completely dependent on the use of technology. From bitcoins to cloud to Internet-of-Things (IoT), new technologies are popping up each day. While these technologies completely change the way we do things, they also bring along threats with them. Attackers discover new and innovative ways to manipulate these technologies for fun and profit! This is a matter of concern for thousands of organizations and businesses around the world. Organizations worldwide are deeply concerned about keeping their data safe. Protecting data is certainly important, however, testing whether adequate protection mechanisms have been put to work is also equally important. Protection mechanisms can fail, hence testing them before someone exploits them for real is a challenging task. Having said this, vulnerability assessment and penetration testing have gained high importance and are now trivially included in all compliance programs. With the vulnerability assessment and penetration testing done in the right way, organizations can ensure that they have put in place the right security controls, and they are functioning as expected!

Vulnerability assessment and penetration testing are two of the most common words that are often used interchangeably. However, it is important to understand the difference between the two. To understand the exact difference, let's consider a real-world scenario:

A thief intends to rob a house. To proceed with his robbery plan, he decides to recon his robbery target. He visits the house (that he intends to rob) casually and tries to gauge what security measures are in place. He notices that there is a window at the backside of the house that is often open, and it's easy to break in. In our terms, the thief just performed a vulnerability assessment. Now, after a few days, the thief actually went to the house again and entered the house through the backside window that he had discovered earlier during his recon phase. In this case, the thief performed an actual penetration into his target house with the intent of robbery.

This is exactly what we can relate to in the case of computing systems and networks. One can first perform a vulnerability assessment of the target in order to assess overall weaknesses in the system and then later perform a planned penetration test to practically check whether the target is vulnerable or not. Without performing a vulnerability assessment, it will not be possible to plan and execute the actual penetration.

While most vulnerability assessments are non-invasive in nature, the penetration test could cause damage to the target if not done in a controlled manner. Depending on the specific compliance needs, some organizations choose to perform only a vulnerability assessment, while others go ahead and perform a penetration test as well.

Penetration testing is not just about running a set of a few automated tools against your target. It's a complete process that involves multiple stages, and each stage is equally important for the success of the project. Now, for performing all tasks throughout all stages of penetration testing, we would need to use various different tools and might need to perform some tasks manually. Then, at the end, we would need to combine results from so many different tools together in order to produce a single meaningful report. This is certainly a daunting task. It would have been really easy and time-saving if one single tool could have helped us perform all the required tasks for penetration testing. This exact need is satisfied by a framework such as Metasploit.

The birth of Metasploit dates back to 14 years ago, when H.D Moore, in 2003, wrote a portable network tool using Perl. By 2007, it was rewritten in Ruby. The Metasploit project received a major commercial boost when Rapid7 acquired the project in 2009. Metasploit is essentially a robust and versatile penetration testing framework. It can literally perform all tasks that are involved in a penetration testing life cycle. With the use of Metasploit, you don't really need to reinvent the wheel! You just need to focus on the core objectives; the supporting actions would all be performed through various components and modules of the framework. Also, since it's a complete framework and not just an application, it can be customized and extended as per our requirements.

Metasploit is, no doubt, a very powerful tool for penetration testing. However, it's certainly not a magic wand that can help you hack into any given target system. It's important to understand the capabilities of Metasploit so that it can be leveraged optimally during penetration testing.

While the initial Metasploit project was open source, after the acquisition by Rapid7, commercial grade versions of Metasploit also came into existence. For the scope of this book, we'll be using the Metasploit Framework edition.

There are literally tons of tools available for performing various tasks related to penetration testing. However, most of the tools serve only one unique purpose. Unlike these tools, Metasploit is the one that can perform multiple tasks throughout the penetration testing life cycle. Before we check the exact use of Metasploit in penetration testing, let's have a brief overview of various phases of penetration testing. The following diagram shows the typical phases of the penetration testing life cycle:

1. Information Gathering: Though the Information Gathering phase may look very trivial, it is one of the most important phases for the success of a penetration testing project. The more you know about your target, the more the chances are that you find the right vulnerabilities and exploits to work for you. Hence, it's worth investing substantial time and efforts in gathering as much information as possible about the target under the scope. Information gathering can be of two types, as follows:
   • Passive information gathering: Passive information gathering involves collecting information about the target through publicly available sources such as social media and search engines. No direct contact with the target is made.
   • Active information gathering: Active information gathering involves the use of specialized tools such as port scanners to gain information about the target system. It involves making direct contact with the target system, hence there could be a possibility of the information gathering attempt getting noticed by the firewall, IDS, or IPS in the target network.
2. Enumeration: Using active and/or passive information gathering techniques, one can have a preliminary overview of the target system/network. Moving further, enumeration allows us to know what the exact services running on the target system (including types and versions) are and other information such as users, shares, and DNS entries. Enumeration prepares a clearer blueprint of the target we are trying to penetrate.
3. Gaining Access: Based on the target blueprint that we obtained from the information gathering and enumeration phase, it's now time to exploit the vulnerabilities in the target system and gain access. Gaining access to this target system involves exploiting one or many of the vulnerabilities found during earlier stages and possibly bypassing the security controls deployed in the target system (such as antivirus, firewall, IDS, and IPS).
4. Privilege Escalation: Quite often, exploiting a vulnerability on the target gives limited access to the system. However, we would want complete root/administrator level access into the target in order to gain most out of our exercise. This can be achieved using various techniques to escalate privileges of the existing user. Once successful, we can have full control over the system with highest privileges and can possibly infiltrate deeper into the target.

5. Maintaining Access: So far, it has taken a lot of effort to gain a root/administrator level access into our target system. Now, what if the administrator of the target system restarts the system? All our hard work will be in vain. In order to avoid this, we need to make a provision for persistent access into the target system so that any restarts of the target system won't affect our access.
6. Covering Tracks: While we have really worked hard to exploit vulnerabilities, escalate privileges, and make our access persistent, it's quite possible that our activities could have triggered an alarm on the security systems of the target system. The incident response team may already be in action, tracing all the evidence that may lead back to us. Based on the agreed penetration testing contract terms, we need to clear all the tools, exploits, and backdoors that we uploaded on the target during the compromise.

Interestingly enough, Metasploit literally helps us in all penetration testing stages listed previously.

The following table lists various Metasploit components and modules that can be used across all stages of penetration testing:

We'll gradually cover all previous components and modules as we progress through the book.

So far we have seen that Metasploit is really a powerful framework for penetration testing. However, it can be made even more useful if integrated with some other tools. This section covers a few tools that compliment Metasploit's capability to perform more precise penetration on the target system.

Nessus is a product from Tenable Network Security and is one of the most popular vulnerability assessment tools. It belongs to the vulnerability scanner category. It is quite easy to use, and it quickly finds out infrastructure-level vulnerabilities in the target system. Once Nessus tells us what vulnerabilities exist on the target system, we can then feed those vulnerabilities to Metasploit to see whether they can be exploited for real.

Its official website is https://www.tenable.com/. The following image shows the Nessus homepage:

The following are the different OS-based installation steps for Nessus:

• Installation on Windows:
  1. Navigate to the URL https://www.tenable.com/products/nessus/select-your-operating-system.
  2. Under the Microsoft Windows category, select the appropriate version (32-bit/64-bit).
  3. Download and install the msi file.
  4. Open a browser and navigate to the URL https://localhost:8834/.
  5. Set a new username and password to access the Nessus console.
  6. For registration, click on the registering this scanner option.
  7. Upon visiting http://www.tenable.com/products/nessus/nessus-plugins/obtain-an-activation-code, select Nessus Home and enter your details for registration.
  8. Enter the registration code that you receive on your email.
• Installation on Linux (Debian-based):
  1. Navigate to the URL https://www.tenable.com/products/nessus/select-your-operating-system.
  2. Under the Linux category, Debian 6,7,8 / Kali Linux 1, select the appropriate version (32-bit/AMD64).
  3. Download the file.
  4. Open a terminal and browse to the folder where you downloaded the installer (.deb) file.
  5. Type the command dpkg -i <name_of_installer>.deb.
  6. Open a browser and navigate to the URL https://localhost:8834/.
  7. Set a new username and password to access the Nessus console.
  8. For registration, click on the registering this scanner option.
  9. Upon visiting http://www.tenable.com/products/nessus/nessus-plugins/obtain-an-activation-code, select Nessus Home and enter your details for registration.
  10. Enter the registration code that you receive on your email.

NMAP (abbreviation for Network Mapper) is a de-facto tool for network information gathering. It belongs to the information gathering and enumeration category. At a glance, it may appear to be quite a small and simple tool. However, it is so comprehensive that a complete book could be dedicated on how to tune and configure NMAP as per our requirements. NMAP can give us a quick overview of what all ports are open and what services are running in our target network. This feed can be given to Metasploit for further action. While a detailed discussion on NMAP is out of the scope for this book, we'll certainly cover all the important aspects of NMAP in the later chapters.

Its official website is https://nmap.org/. The following screenshot shows a sample NMAP scan:

While the most common way of accessing NMAP is through the command line, NMAP also has a graphical interface known as Zenmap, which is a simplified interface on the NMAP engine, as follows:

The following are the different OS-based installation steps for NMAP:

• Installation on Windows:
  1. Navigate to site https://nmap.org/download.html.
  2. Under the Microsoft Windows Binaries section, select the latest version (.exe) file.
  3. Install the downloaded file along with WinPCAP (if not already installed).

• Installation on Linux (Debian-based): NMAP is by default installed in Kali Linux; however, if not installed, you can use the following command to install it:

root@kali:~#apt-get install nmap

w3af is an open-source web application security scanning tool. It belongs to the web application security scanner category. It can quickly scan the target web application for common web application vulnerabilities, including the OWASP Top 10. w3af can also be effectively integrated with Metasploit to make it even more powerful.

Its official website is http://w3af.org/. We can see the w3af console for scanning web application vulnerabilities in the following image:

The following are the various OS-based installation steps for w3af:

• Installation on Windows: w3af is not available for the Windows platform
• Installation on Linux (Debian-based): w3af is by default installed on Kali Linux; however, if not installed, you can use the following command to install it:

root@kali:~# apt-get install w3af

Armitage is an exploit automation framework that uses Metasploit at the backend. It belongs to the exploit automation category. It offers an easy-to-use user interface for finding hosts in the network, scanning, enumeration, finding vulnerabilities, and exploiting them using Metasploit exploits and payloads. We'll have a detailed overview of Armitage later in this book.

Its official website is http://www.fastandeasyhacking.com/index.html. We can see the Armitage console for exploit automation in the following screenshot:

The following are the various OS-based installation steps for Armitage:

• Installation on Windows: Armitage is not supported on Windows
• Installation on Linux (Debian-based): Armitage is by default installed on Kali Linux; however, if not installed, you can use the following command to install it:

root@kali:~# apt-get install armitage

Now that we have got a high-level overview of what Metasploit is all about, its applicability in penetration testing, and supporting tools, we'll browse through the installation and environment setup for Metasploit in the next chapter.

You can try the following exercises:

• Visit Metasploit's official website and try to learn about the differences in various editions of Metasploit
• Try to explore more on how Nessus and NMAP can help us during a penetration test.