Extracting Faces

In this chapter, we will start our hands-on activities with the code. To begin with, we’ll cover the process of extracting faces.

Extracting faces is a series of steps in a process that involves many different stages, but it’s the first discrete stage in creating a deepfake. This chapter will first talk about how to run the face extraction script, then we will go hands-on with the code, explaining what each part does.

In this chapter, we will cover the following key sections:

• Getting image files from a video
• Running extract on frame images
• Getting hands-on with the code

Technical requirements

To proceed, we recommend that you download the code from the GitHub repo at https://github.com/PacktPublishing/Exploring-Deepfakes and follow the instructions in the readme.md file to install Anaconda and to create a virtual environment with all required libraries to run the scripts. The repo will also contain any errata or updates that have happened since the book was published, so please check there for any updates.

Getting image files from a video

Videos are not designed for frame-by-frame access and can cause problems when processed out of order. Accessing a video file is a complicated process and not good for a beginner-level chapter like this. For this reason, the first task is to convert any videos that you want to extract from into individual frames. The best way to do this is to use FFmpeg. If you followed the installation instructions in the Technical requirements section, you will have FFmpeg installed and ready to use.

Let’s begin the process.

Tip

When you see code or command examples such as those present here with text inside of curly brackets, you should replace that text with the information that is explained in the brackets. For example, if it says cd {Folder with video} and the video is in the c:\Videos\ folder, then you should enter cd c:\Videos.

Place the video into a folder, then open an Anaconda prompt and enter the following commands:

cd {Folder with...

Running extract on frame images

To run the extract process on a video, you can run the extract program from the cloned git repository folder. To do this, simply run the following in an Anaconda prompt:

cd {Folder of the downloaded git repo}\
python C5-face_detection.py {Folder of frame images}

This will run the face extraction process on each of the images in the folder and put the extracted images into the face_images folder, which (by default) will be inside the folder of frame images. This folder will contain three types of files for each face detected, along with a file containing all alignments.

face_alignments.json

There will just be one of these files. It is a JSON-formatted file containing the landmark positions and warp matrix for every face found in the images. This file is human-readable like any JSON file and can be read or edited (though it’s probably not something that you’d do manually).

face_landmarks_{filename}_{face number}.png

This...

Getting hands-on with the code

Now it’s time to get into the code. We’ll go over exactly what C5-face_detection.py does and why each option was chosen. There are five main parts to the code: initialization, image preparation, face detection, face landmarking/aligning, and masking.

Initialization

Let’s begin.

Author’s note

Formatting for easy reading in a book requires modifying the spacing in the samples. Python, however, is whitespace sensitive and uses spacing as a part of the language syntax. This means that copying code from this book will almost definitely contain the wrong spacing. For this reason, we highly recommend pulling the code from the Git repository for the book at https://github.com/PacktPublishing/Exploring-Deepfakes if you plan on running it.

1. First, we import all required libraries:

   import os
   import torch
   import cv2
   import json_tricks
   import numpy as np
   from tqdm import tqdm
   from argparse import ArgumentParser
   from face_alignment...

Summary

Extraction is the first step of the training process. In this chapter, we examined the data that we will need in later steps as well as the process of extracting the required training data from the source images. We went hands-on with the process, using multiple AIs to detect and landmark faces and generate a mask, as well as the necessary steps to process and save that data.

The C5-face_detection.py file can process a directory of images. So, we covered how to convert a video into a directory of images and how to process that directory through the script. The script creates all the files you need for training and some interesting debug images that let you visualize each of the processes the detector uses to process the images. We then looked at the entire process, line by line, so that we knew exactly what was going on inside that script, learning not just what was being output, but exactly how that output was created.

After finishing the detection process, you can go...

Exercises

1. We used pre-existing libraries for face detection, landmarking, and aligning landmarks. There are other libraries that offer similar functionality. Not all libraries work the same way, and implementing the differences is an extremely useful exercise. Try replacing the face_alignment library with another library for detecting faces, such as https://github.com/timesler/facenet-pytorch or https://github.com/serengil/deepface. Open source has lots of useful libraries but learning the differences and when to use one over another can be difficult, and converting between them can be a useful practice.
2. We used 2D landmarks for alignment in this chapter, but there may be a need for 3D landmarks instead. Try replacing the following:

   face_aligner = FaceAlignment(LandmarksType._2D,
     device=device, verbose=False)

with:

face_aligner = FaceAlignment(LandmarksType._3D,
  device=device, verbose=False)

and adjust the rest of the process accordingly...