ELF anti-debugging and packing techniques

In the next chapter, Breaking ELF Software Protection, we will discuss the ins and outs of software encryption and packing with ELF executables. Viruses and malware are very commonly encrypted or packed with some type of protection mechanism, which can also include anti-debugging techniques to make analyzing the binary very difficult. Without giving a complete exegesis on the subject, here are some common anti-debugging measures taken by ELF binary protectors that are commonly used to wrap around malware.

The PTRACE_TRACEME technique

This technique takes advantage of the fact that a program can only be traced by one process at a time. Almost all debuggers use ptrace, including GDB. The idea is that a program can trace itself so that no other debugger can attach.

Illustration 4.9 – an anti-debug with PTRACE_TRACEME example

void anti_debug_check(void)
{
  if (ptrace(PTRACE_TRACEME, 0, 0, 0) < 0) {
    printf("A debugger is attached, but not for long...