Can You Find Out Which Movie Is a Deepfake?

In the previous chapters, we explored various data formats: tabular, geospatial, text, image, and acoustic, while working with Kaggle datasets, learning about shapefile visualization, building models for image or text classification, and acoustic signal analysis.

In this chapter, we will introduce video data analysis. We will start by describing a Kaggle competition, Deepfake Detection Challenge. This competition challenged the participants to classify which videos were generated artificially to create realistic fake content convincingly. We will continue by quickly exploring the most used video formats, followed by introducing two utility scripts used for our data analysis. First, a utility script with functions for manipulating video content, i.e., reading, visualizing images from videos, and playing video files. Second, a utility script with functions for body, face, and face element detection. We will continue with metadata exploration...

Introducing the competition

In this chapter, we examine data from the well-known Kaggle competition the Deepfake Detection Challenge (DFDC). The competition, detailed in Reference 1, commenced on December 11, 2019, and concluded on March 31, 2020. It attracted 2,265 teams comprising 2,904 participants, who collectively made 8,951 submissions. Competitors vied for a total prize pool of $1,000,000, with the first prize being $500,000.

The event was a collaborative effort involving AWS, Facebook, Microsoft, the Partnership on AI’s Media Integrity Steering Committee, and various academic entities. At the time, there was a widespread agreement among tech industry leaders and academics on the technical complexity and rapidly changing nature of media content manipulation. The competition’s aim was to encourage global researchers to devise innovative and effective technologies to detect deepfakes and media manipulation. Unlike later competitions that focused on code, this...

Introducing competition utility scripts

Let’s begin by grouping the Python modules with reusable functions for video manipulation in two Kaggle utility scripts. The first utility script groups functions to load and display images from videos or play video files. The second one is geared toward object detection in videos – more specifically, to detect human faces and bodies – using a few alternative methods.

Video data utils

We developed a utility script to assist us in the manipulation of video data. Let’s introduce one utility script that we will use in the notebook associated with the current chapter to read video data, as well as visualize frames from a video file.

The video_utils utility script includes functions to load, transform, and display images from videos. Additionally, it also contains a function to play video content. For video manipulation, we will use the OpenCV library. OpenCV is an open-source computer vision library widely...

Metadata exploration

We start by importing the utility functions and classes from the utility scripts for data quality, plot utils, video utils, and face object detection. The following code block shows what we import from the utility scripts:

from data_quality_stats import missing_data, unique_values, most_frequent_values
from plot_style_utils import set_color_map, plot_count
from video_utils import display_image_from_video, display_images_from_video_list, play_video
from face_object_detection import CascadeObjectDetector, FaceObjectDetector
from face_detection_mtcnn import MTCNNFaceDetector

After we load the data files (the train and test samples), we are ready to start our analysis. The following code block checks the types of files in TRAIN_SAMPLE_FOLDER:

train_list = list(os.listdir(os.path.join(DATA_FOLDER, TRAIN_SAMPLE_FOLDER)))
ext_dict = []
for file in train_list:
    file_ext = file.split('.')[1]
    if (file_ext not in ext_dict):
        ext_dict...

Video data exploration

In this section, we will visualize a few samples of files, and then we will begin performing object detection to try to capture the features from the images that might have some anomalies when processed to create deepfakes. These are mostly the eyes, mouths, and figures.

We will start by visualizing sample files, both genuine images and deepfakes. We will then apply the first algorithm introduced previously for face, eye, and mouth detection, the one based on Haar cascade. We then follow with the alternative algorithm, based on MTCNN.

Visualizing sample files

The following code block selects a few video files from the set of fake videos and then visualizes an image capture from them, using the display_image_from_video function from the utility script video_utils:

fake_train_sample_video = list(meta_train_df.loc[meta_train_df.label=='FAKE'].sample(3).index)
for video_file in fake_train_sample_video:
    display_image_from_video(os.path...

Summary

In this chapter, we began by introducing a series of utility scripts, which are reusable Python modules on Kaggle designed for video data manipulation. One such script, video_utils, is used to visualize images from videos and play them. Another script, face_object_detection, utilizes Haar cascade models for face detection.

The third script, face_detection_mtcnn, employs MTCNN models to identify faces and key points such as the eyes, nose, and mouth. We then examined the metadata and video data from the DFDC competition dataset. In this dataset, we applied the aforementioned face detection methods to images from training and test videos, finding the MTCNN model approach to be more robust and accurate, with fewer false positives.

As we near the conclusion of our exploration of data, we will reflect on our journey through various data formats, including tabular, text, image, sound, and now video. We’ve delved into numerous Kaggle datasets and competition datasets...

References

1. Deepfake Detection Challenge, Kaggle competition, Identify videos with facial or voice manipulations: https://www.kaggle.com/competitions/deepfake-detection-challenge
2. Haar Cascades for Face Detection, Kaggle dataset: https://www.kaggle.com/datasets/gpreda/haar-cascades-for-face-detection
3. Serkan Peldek – Face Detection with OpenCV, Kaggle notebook: https://www.kaggle.com/code/serkanpeldek/face-detection-with-opencv/
4. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao – Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks: https://arxiv.org/abs/1604.02878
5. Justin Güse – Face Detection using MTCNN — a guide for face extraction with a focus on speed: https://towardsdatascience.com/face-detection-using-mtcnn-a-guide-for-face-extraction-with-a-focus-on-speed-c6d59f82d49

Join our book’s Discord space

Join our Discord community to meet like-minded people and learn...