A ptrace anti-debugging trick

The ptrace command can be used as an anti-debugging technique. Often when a hacker doesn't want their program to be easily debugged, they include certain anti-debugging techniques. One popular way in Linux is to use ptrace with the PTRACE_TRACEME request so that it traces the process of itself.

Remember that a process can only have one tracer at a time, so if a process is already being traced and a debugger tries to attach using ptrace, it says Operation not permitted. PTRACE_TRACEME can also be used to check whether your program is already being debugged. You can use the code in the following section to check this.

Is your program being traced?

Let's take a look at a code snippet that will use ptrace to find out whether your program is already being traced:

if (ptrace(PTRACE_TRACEME, 0) < 0) {
printf("This process is being debugged!!!\n");
exit(1);
}

The preceding code works because it should only fail if the program is already being traced. So, if ptrace returns...