Swapping the Face Back into the Video

In this chapter, we’ll complete the deepfake process by converting the videos to swap faces using the models trained in the last chapter.

Conversion is the last step of deepfaking, and it is the part that actually puts the new face onto the existing video. This requires you to already have a video that you have fully processed through the extraction process in Chapter 5, Extracting Faces, and uses a trained model from Chapter 6, Training a Deepfake Model.

We will cover the following topics in this chapter:

• Preparing to convert video
• Getting hands-on with the convert code
• Creating the video from images

Technical requirements

For this chapter, you’ll need a conda environment setup. If you set this up in earlier chapters, the same conda environment will work fine. To get into the conda environment, you can run the following command:

conda activate deepfakes

If you have not created a conda environment to run the code, it’s recommended that you go to the Git repository and follow the instructions there. You can find the full repository at https://github.com/PacktPublishing/Exploring-Deepfakes.

Preparing to convert video

Conversion is not just a “one-and-done” script. It requires you to have turned a video into a series of frames and run C5-face_detection.py on those frames. This gets the data that you need for the conversion process in the right form. The conversion process will require the full extraction of every frame, as well as the face_alignments.json file that is generated by the extract process:

Figure 7.1 – Example of a folder that has been extracted already. Note the face_images folder created by the extract process

If you haven’t done the extract process on the video you want to convert, then you should go back to Chapter 5, Extracting Faces, and extract the video.

We need to do this because this is how the model knows which faces to convert. AI can detect all faces in a frame but won’t know which ones should be swapped, meaning that all faces will be swapped. By running the extract process and...

Getting hands-on with the convert code

Like the rest of the chapters in this section, we’ll be going through the code line by line to talk about how it works and what it’s doing.

Initialization

Here we will initialize and prepare the code to run the convert process:

1. Like all Python code, we’ll start with the imports:

   import os
   from argparse import ArgumentParser
   import json_tricks
   import torch
   import cv2
   import numpy as np
   from tqdm import tqdm
   import face_alignment
   from face_alignment.detection.sfd import FaceDetector
   from face_alignment import FaceAlignment, LandmarksType
   from lib.bisenet import BiSeNet
   from lib.models import OriginalEncoder, OriginalDecoder

These libraries are all ones we’ve already seen in previous chapters. This is because the conversion process is not really doing anything too different from what we’ve done before. We’ll see that as we go through the code to covert the face back into the original images...

Creating the video from images

The conversion code included produces swapped images, but if we want to create a video, we’ll need to combine the output into a video file. There are multiple options here, depending on what you want to include:

• The following is for including just the images:

  ffmpeg -i {path_to_convert}\%05d.png Output.mp4

This command line will convert all the frames into a video with some default options. The Output.mp4 file will include the frames but won’t include any audio and will be at a default frame rate of 25 frames per second. This will be close enough to accurate for videos that came from film sources, such as Blu-rays or DVDs. If the video looks too fast or too slow, then your frame rate is incorrect, and you should look at the next option instead to match the correct frame rate.

• Including the images at a specific frame rate:

  ffmpeg -framerate {framerate} -i {path_to_convert}\%05d.png
    Output.mp4

This command...

Summary

In this chapter, we ran the convert process on a folder full of images, replacing the faces using a trained model. We also turned the images back into a video, including changes to account for frame rate and copying audio.

We started by going over how to prepare a video for conversion. The convert process requires data created by the extract process from Chapter 5, Extracting Faces, and a trained AI model from Chapter 6, Training a Deepfake Model. With all the parts from the previous chapters, we were ready to convert.

We then walked through the code for the conversion process. This involved looking at the initialization, where we covered getting the Python script ready to operate. We then loaded the AI models and got them set up to work on a GPU if we have one. Next, we got the data ready for us to convert the faces in each frame. Finally, we ran two nested loops, which processed every face in every frame, swapping them to the other face. This part gave us a folder filled...

Exercises

1. We use the mask to cut out the swapped face from the rest of the image but then copy it over to the aligned face. This means that the areas of the aligned image that aren’t the face also get a lower resolution. One way to fix this would be to apply the mask to the original image instead of the aligned image. To do this, you’ll need to call cv2.warpAffine separately for the mask and the aligned image, then use the mask to get just the face copied over. You may want to view the documentation for OpenCV’s warpAffine at https://docs.opencv.org/3.4/d4/d61/tutorial_warp_affine.html.

Be sure to account for the fact that OpenCV’s documentation is based on the C++ implementation, and things can be a bit different in the Python library. The tutorial pages have a Python button that lets you switch the tutorial to using the Python libraries.

2. We rely on pre-extracted faces in order to convert. This is because a lot of the data is already...