Augmented Reality

Augmented reality is one of the hottest trends at present. The concept of augmented reality can be defined as an improved version of reality, where the view of the real world is enhanced with superimposed computer-generated elements (for example, images, videos, or 3D models, among others). In order to overlay and integrate digital information—augmenting reality, different types of techniques can be used, mainly location-based and recognition-based approaches.

In this chapter, we will cover the main concepts related to augmented reality and we will also code some interesting applications in order to see the potential of this technology. In this chapter, you will learn how to build your first augmented reality applications. At the end of this chapter, you will have the knowledge to create your augmented reality applications using OpenCV.

The main sections...

Technical requirements

The technical requirements are listed here:

• Python and OpenCV
• Python-specific IDE
• NumPy and Matplotlib packages
• Git client

Further details about how to install these requirements are commented on in Chapter 1, Setting Up OpenCV.

The GitHub repository for Mastering OpenCV 4 with Python, containing all of the supporting project files necessary to work through this book, from the first chapter to the last, can be accessed in the next URL: https://github.com/PacktPublishing/Mastering-OpenCV-4-with-Python/.

An introduction to augmented reality

Location-based and recognition-based augmented reality are the two main types of augmented reality. Both types try to derive where the user is looking. This information is key in the augmented reality process, and relies on properly calculating the camera pose estimation. In order to accomplish this task, the two types are briefly described as follows:

• Location-based augmented reality relies on detecting the user's location and orientation by reading data from several sensors, that are very common in smartphone devices (for example, GPS, digital compass, and accelerometer) to derive where the user is looking. This information is used to superimpose computer-generated elements on the screen.
• On the other hand, recognition-based augmented reality uses image processing techniques to derive where the user is looking. Obtaining the camera...

Markerless-based augmented reality

As commented before, camera pose estimation can be derived from images to find the correspondences between known points in the environment and their camera projections. In this section, we will see how to extract features from the images to derive the camera pose. Based on these features and their matching, we will see how to finally derive camera pose estimation, which is then used to overlay and integrate digital information.

Feature detection

A feature can be described as a small patch in the image, which is invariant (as much as possible) to image scaling, rotation, and illumination. This way, the same feature can be detected from different images of the same scene with different perspectives...

Marker-based augmented reality

In this section, we are going to see how marker-based augmented reality works. There are many libraries, algorithms, or packages that you can use to both generate and detect markers. In this sense, one that provides state-of-the-art performance in detecting markers is ArUco.

ArUco automatically detects the markers and corrects possible errors. Additionally, ArUco proposes a solution to the occlusion problem by combining multiple markers with an occlusion mask, which is calculated by color segmentation.

As previously commented, pose estimation is a key process in augmented reality applications. Pose estimation can be performed based on markers. The main benefit of using markers is that they can be both efficiently and robustly detected in the image where the four corners of the marker can be accurately derived. Finally, the camera pose can be obtained...

Snapchat-based augmented reality

In this section, we are going to see how to create some funny Snapchat-based filters. In this case, we are going to create two filters. The first one overlays a big moustache between the nose and mouth on the detected face. The second one overlays a pair of glasses on the detected face.

In the following subsections, you will see how to achieve this functionality.

Snapchat-based augmented reality OpenCV moustache overlay

In the snapchat_augmeted_reality_moustache.py script, we overlay a moustache on the detected face. Images are continuously captured from the webcam. We have also included the possibility to use a test image instead of the images captured from the webcam. This can be useful in...

QR code detection

To complete this chapter, we are going to learn how to detect QR codes in images. This way, QR codes can also be used as markers for our augmented reality applications. The cv2.detectAndDecode() function both detects and decodes a QR code in the image containing the QR code. The image can be grayscale or color (BGR).

This function returns the following:

• An array of vertices of the found QR code is returned. This array can be empty if the QR code is not found.
• The rectified and binarized QR code is returned.
• The data associated with this QR code is returned.

In the qr_code_scanner.py script, we make use of the aforementioned function to detect and decode QR codes. The key points are commented next.

First, the image is loaded, as follows:

image = cv2.imread("qrcode_rotate_45_image.png")

Next, we create the QR code detector with the following code...

Summary

In this chapter, we looked at an introduction to augmented reality. We coded several examples to see how to build both marker and markerless augmented reality applications. Additionally, we saw how to overlay simple models (shapes or images, among others).

As commented previously, to overlay more complex models, PyOpenGL (standard OpenGL bindings for Python) can be used. In this chapter, for the sake of simplification, this library is not tackled.

We have also seen how to create some funny Snapchat-based filters. It should be noted that in Chapter 11, Face Detection, Tracking, and Recognition, more advanced algorithms for both face detection, tracking, and location of facial landmarks will be covered. Therefore, Snapchat-based filters coded in this chapter can be easily modified to include a more robust pipeline to derive the position where both the glasses and the moustache...

Questions

1. Initialize the ORB detector, find keypoints, and compute descriptors in the loaded image image with ORB
2. Draw the previously detected keypoints
3. Create the BFMatcher object and match the descriptors_1 and descriptors_2, which have been previously calculated
4. Sort the matches calculated before and draw the first 20 matches
5. Detect markers using ArUco in the gray_frame image
6. Draw detected markers using ArUco
7. Draw rejected markers using ArUco
8. Detect and decode a QR code contained in the image image

Further reading

The following reference will help you dive deeper into augmented reality:

• Augmented Reality for Developers (2017) (https://www.packtpub.com/web-development/augmented-reality-developers)