Chapter 5. Linux Binary Protection

In this chapter, we are going to explore the basic techniques and motivations for obfuscation of Linux programs. Techniques that obfuscate or encrypt binaries or make them difficult to tamper with are called software protection schemes. By "software protection," we mean binary protection or binary hardening techniques. Binary hardening is not exclusive to Linux; in fact, there are many more products for the Windows OS in this technology genre, and there are definitely more examples to choose from for discussion.

What many people fail to realize is that Linux has a market for this too, although it largely exists for anti-tamper products used by the government. There are also a number of ELF binary protectors that were released over the last decade in the hacker community, several of which paved the way for many of the technologies used today.

An entire book could be dedicated to the art of software protection, and as the author of some of the more recent binary...

ELF binary packers – dumb protectors

A packer is a type of software that is commonly used by malware authors and hackers to compress or encrypt an executable in order to obfuscate its code and data. One very common packer is named UPX (http://upx.sourceforge.net) and is available as a package on most Linux distributions. The original purpose of this type of packer was to compress an executable and make it smaller.

Since the code is compressed, it must have a way to decompress itself before executing in memory—this is where things get interesting, and we will discuss how this works in the Stub mechanics and the userland exec section. At any rate, malware authors have realized that compressing their malware-infected files would evade AV detection due to obfuscation. This led malware/antivirus researchers to develop automated unpackers, which are now used in most, if not all, modern AV products.

Nowadays, the term "packed binary" refers not only to compressed binaries but also to encrypted binaries...

Stub mechanics and the userland exec

First, it is necessary to understand that a software protector is actually made up of two programs:

• Protection phase code: The program that applies the protection to the target binary

• Runtime engine or stub: The program that is merged with the target binary that is responsible for deobfuscation and anti-debugging at runtime

The protector program can vary greatly depending on the types of protection that are being applied to the target binary. Whatever type of protection is being applied to the target binary must be understood by the runtime code. The runtime code (or stub) must know how to decrypt or deobfuscate the binary that it is merged with. In most cases of software protection, there is a relatively simple runtime engine merged with the protected binary; its sole purpose is to decrypt the binary and pass control to the decrypted binary in memory.

This type of runtime engine is not so much an engine—really—and we call it a stub. The stub is generally...

Other jobs performed by protector stubs

In addition to decrypting and loading the embedded executable into memory, which is the userland exec component, the stub may also perform other tasks. It is common for the stub to start anti-debugging and anti-emulation routines that are meant to further protect the binary from being debugged or emulated in order to raise the bar even further so that reverse engineering is even more difficult.

In Chapter 4, ELF Virus Technology – Linux/Unix Viruses, we discussed some anti-debugging techniques used to prevent debugging based on ptrace. This prevents most debuggers, including GDB, from trivially tracing the binary. Later in this chapter, we will summarize the most common anti-debugging techniques used in binary protection for Linux.

Existing ELF binary protectors

Over the years, there have been a few noteworthy binary protectors that were released both publicly and from the underground scene. I will discuss some of the protectors for Linux and give a synopsis of the various features.

DacryFile by the Grugq – 2001

DacryFile is the earliest binary protector that I am aware of for Linux (https://github.com/packz/binary-encryption/tree/master/binary-encryption/dacryfile). This protector is simple but nonetheless clever and works very similarly to ELF parasite infection from a virus. In many protectors, the stub wraps around the encrypted binary, but in the case of DacryFile, the stub is just a simple decryption routine that is injected into the binary that is to be protected.

DacryFile encrypts a binary from the beginning of the .text section to the end of the text segment using RC4 encryption. The decryption stub is a simple program written in asm and C, and it does not have the userland exec functionality; it simply decrypts...

Downloading Maya-protected binaries

For those who are interested in reverse-engineering some simple programs that were protected with Maya's Veil, feel free to download a couple of samples that are available at http://www.bitlackeys.org/maya_crackmes.tgz. This link contains three files: crackme.elf_hardest, crackme.elf_medium, and test.maya.

Anti-debugging for binary protection

Since binary protectors generally encrypt or obfuscate the physical body of a program, static analysis can be extremely difficult and, left to its own devises, will prove to be futile in many cases. Most reverse engineers who are attempting to unpack or break a protected binary will agree that a combination of dynamic analysis and static analysis must be used to gain access to the decrypted body of a binary.

A protected binary has to decrypt itself, or at least the portions of itself that are executing at runtime. Without any anti-debugging techniques, a reverse engineer can simply attach to the process of the protected program and set a breakpoint on the last instruction of the stub (assuming that the stub decrypts the entire executable).

Once the breakpoint is hit, the attacker can look at the code segment for where the protected binary lives and find its decrypted body. This would be extremely simple, and therefore it is very important for good binary...

Resistance to emulation

Often, emulators are used to perform dynamic analysis and reverse engineering tasks on executables. One very good reason for this is that they allow the reverse engineer to easily instrument the control of the execution, and they also bypass a lot of typical anti-debugging techniques. There are many emulators being used out there—QEMU, BOCHS, and Chris Eagles' IDA X86 emulator plugin, to name some. So, countless anti-emulation techniques exist, but some of them are specific to each emulator's particular implementation.

This topic could expand into some very in-depth discussions and move in many directions, but I will keep it limited to my own experience. In my own experimentation with emulation and anti-emulation in the Maya protector, I have learned some generic techniques that should work against at least some emulators. The goal of our binary protector's anti-emulation is to be able to detect when it is being run in an emulator, and if this is true, it should halt...

Obfuscation methods

A binary can be obfuscated or encrypted in many creative ways. Most binary protectors simply protect the entire binary with one or more layers of protection. At runtime, the binary is decrypted and can be dumped from the memory to acquire a copy of the unpacked binary. In more advanced protectors, such as Maya, every single function is encrypted individually, and allows only a single function to be decrypted at any given time.

Once a binary is encrypted, it must, of course, store the encryption keys somewhere. In the case of Maya (discussed earlier), a custom heap implementation that itself uses encryption to store encryption keys was designed. At some point, it would seem that a key has to be exposed (such as the key used to decrypt another key), but special techniques such as white-box cryptography can be used to make these final keys extremely obfuscated. If assistance from the kernel is used in a protector, then it is possible to store the key outside of the binary...

Protecting control flow integrity

A protected binary should aim to protect the program during runtime (the process itself) just as much as—if not more than—the binary at rest on the disk. Runtime attacks can generally be classified into two types:

• Attacks based on ptrace

• Vulnerability-based attacks

Attacks based on ptrace

The first variety, ptrace based attacks, also falls under the category of debugging a process. As already discussed, a binary protector wants to make ptrace based debugging very difficult for a reverse engineer. Aside from debugging, however, there are many other attacks that could potentially help break a protected binary, and it is important to know and understand what some of these are in order to give further clarification as to why a binary protector wants to protect a running process from ptrace.

If a protector has gone so far that it is able to detect breakpoint instructions (and therefore make debugging more difficult) but is not able to protect itself from being traced...

Other resources

Writing only one chapter on binary protection is not nearly comprehensive enough on its own to teach you all about this one subject. The other chapters in this book complement each other, however; when combined together, they will help you get to deeper levels of understanding. There are many good resources on this subject, some of which have already been mentioned.

One resource in particular, written by Andrew Griffith, is highly recommended for reading. This paper was written over a decade ago but describes many of the techniques and practices that are still very pertinent to the binary protectors of today:

http://www.bitlackeys.org/resources/binary_protection_schemes.pdf

This paper was followed by a talk given at a later date, and the slides can be found here:

http://2005.recon.cx/recon2005/papers/Andrew_Griffiths/protecting_binaries.pdf

Summary

In this chapter, we revealed the inner workings of basic binary protection schemes for Linux binaries, and discussed the various features of existing binary protectors that have been released for Linux over the last decade.

In the next chapter, we will be exploring things from the opposite angle and begin looking at ELF binary forensics in Linux.