Chapter 11. Anti-analysis Tricks

Anti-debugging, anti-virtual-machine (VM), anti-emulation, and anti-dumping are all tricks that attempt to analysis put a halt to an analysis. In this chapter, we will try to show the concepts of these anti-analysis methods. To help us identify these codes, we will explain the concept and show the actual disassembly codes that makes it work. Being able to identify these tricks will help us to avoid them. With initial static analysis, we would be able to skip these codes.

In this chapter, we will achieve the following learning outcomes:

• Identifying anti-analysis tricks
• Learning how to overcome anti-analysis tricks

Anti-debugging tricks

Anti-debugging tricks are meantto ensure that the codes are not working under the influence of a debugger. Say we have a program with an anti-debugging code in it. The behavior of the program is just as if it were running without an anti-debugging code. The story becomes different, however, when the program is being debugged. While debugging, we encounter code that goes straight to exiting the program or jumps into code that doesn't make sense. This process is illustrated in the following diagram:

Developing anti-debugging code requires understanding the traits of the program and the system, both when normally running and when being debugged. For example, the Process Environment Block (PEB) contains a flag that is set when a program is being run under a debugger. Another popular trick is to use a Structured Exception Handler (SEH) to continue code that forces an error exception while debugging. To better understand how these work, let's discuss these tricks in a little...

Anti-VM tricks

This trick's aim is to exit the program when it identifies that it is running in a virtualized environment. The most typical way to identify being in a VM is to check for specific virtualization software artifacts installed in the machine. These artifacts may be located in the registry or a running service. We have listed a few specific artifacts that can be used to identify being run inside a VM. 

VM running process names

The easiest way for a program to determine whether it is in a VM is by identifying known file names of running processes. Here's a list for each of the most popular pieces of VM software:

Virtualbox	VMWare	QEMU	Parallels	VirtualPC
vboxtray.exevboxservice.exevboxcontrol.exe	vmtoolsd.exevmwaretray.exevmwareuserVGAuthService.exevmacthlp.exe	qemu-ga.exe	prl_cc.exeprl_tools.exe	vmsrvc.exevmusrvc.exe

Existence of VM files and directories

Identifying the existence of at least one of the VM software's files can tell if the program is running in a virtual machine. The following...

Anti-emulation tricks

Anti-emulation or anti-automated analysis are methods employed by a program to prevent moving further in its code if it identifies that it is being analyzed. The behavior of a program can be logged and analyzed using automated analysis tools such as Cuckoo Sandbox, Hybrid Analysis, and ThreatAnalyzer. The concept of these tricks is in being able to determine that the system in which a program is running is controlled and was set up by a user.

Here are some things that distinguish a user-controlled environment and an automated analysis controlled system from each other:

• A user-controlled system has mouse movement.
• User controlled systems can include a dialog box that waits for a user to scroll down and then click on a button.
• The setup of an automated analysis system has the following attributes:
  • A low amount of physical memory
  • A low disk size
  • The free space on the disk may be nearly depleted
  • The number of CPUs is only one
  • The screen size is too small

Simply setting up a task...

Anti-dumping tricks

This method does not stop dumping memory to a file. This trick instead prevents the reverser from easily understanding the dumped data. Here are some examples of how this could be applied:

• Portions of the PE header have been modified, so that the process dump gives the wrong properties.

• Portions of PEB, such as SizeOfImage, have been modified, so that the process dumping tool dumps wrong.

• Dumping is very useful for seeing decrypted data. Anti-dumping tricks would re-encrypt the decrypted code or data after use.

To overcome this trick, we can either identify or skip the code that modifies data. For re-encryption, we can also skip the code that re-encrypts, to leave it in a decrypted state.

Summary

Malware have been evolving by adding new techniques to evade anti-virus and reverse engineering.  These techniques include process hollowing, process injection, process doppelganging, code anti-debugging, and anti-analysis.  Process hollowing and process doppelganging techniques basically overwrites the image of a legit process with a malicious image.  This masks the malicious program with a legit process.  Process injection, on the other hand, inserts and runs code in a remote process space.

Anti-debugging, anti-analysis, and the other tricks discussed in this chapter are obstacles for reverse engineering. But knowing the concept for these tricks enables us to overcome them. Doing static analysis with deadlisting, we can identify and then skip the tricky code, or in the case of SEH, place a breakpoint at the handler.

We discussed anti-debugging tricks and their technique of using errors to cause exceptions and hold the rest of its code at the handler. We also discussed other tricks...

Anti-emulation tricks

Anti-emulation or anti-automated analysis are methods employed by a program to prevent moving further in its code if it identifies that it is being analyzed. The behavior of a program can be logged and analyzed using automated analysis tools such as Cuckoo Sandbox, Hybrid Analysis, and ThreatAnalyzer. The concept of these tricks is in being able to determine that the system in which a program is running is controlled and was set up by a user.

Here are some things that distinguish a user-controlled environment and an automated analysis controlled system from each other:

• A user-controlled system has mouse movement.
• User controlled systems can include a dialog box that waits for a user to scroll down and then click on a button.
• The setup of an automated analysis system has the following attributes:
  • A low amount of physical memory
  • A low disk size
  • The free space on the disk may be nearly depleted
  • The number of CPUs is only one
  • The screen size is too small

Simply setting up a task...

Anti-dumping tricks

This method does not stop dumping memory to a file. This trick instead prevents the reverser from easily understanding the dumped data. Here are some examples of how this could be applied:

• Portions of the PE header have been modified, so that the process dump gives the wrong properties.

• Portions of PEB, such as SizeOfImage, have been modified, so that the process dumping tool dumps wrong.

• Dumping is very useful for seeing decrypted data. Anti-dumping tricks would re-encrypt the decrypted code or data after use.

To overcome this trick, we can either identify or skip the code that modifies data. For re-encryption, we can also skip the code that re-encrypts, to leave it in a decrypted state.

Summary

Malware have been evolving by adding new techniques to evade anti-virus and reverse engineering.  These techniques include process hollowing, process injection, process doppelganging, code anti-debugging, and anti-analysis.  Process hollowing and process doppelganging techniques basically overwrites the image of a legit process with a malicious image.  This masks the malicious program with a legit process.  Process injection, on the other hand, inserts and runs code in a remote process space.

Anti-debugging, anti-analysis, and the other tricks discussed in this chapter are obstacles for reverse engineering. But knowing the concept for these tricks enables us to overcome them. Doing static analysis with deadlisting, we can identify and then skip the tricky code, or in the case of SEH, place a breakpoint at the handler.

We discussed anti-debugging tricks and their technique of using errors to cause exceptions and hold the rest of its code at the handler. We also discussed other tricks...