Instant OpenCV for iOS: Learn how to build real-time computer vision applications for the iOS platform using the OpenCV library

Apple has established a very rich and sound ecosystem for developers, where each component is tightly integrated with others. Once you're familiar with its basic rules and principles, you'll be able to switch between different types of projects easily. But it may take some time to familiarize yourself with the tools and frameworks. And the very first prerequisite for iOS development is a Mac OS X workstation or laptop, and you cannot use other operating systems. It is also highly recommended to use the latest available version of the OS and tools, as some new features are not backported to older versions. Currently, the latest version is the Mac OS X 10.8, also known as Mountain Lion.

Secondly, to run some examples from this book, you will definitely need an iOS device, because iOS Simulator lacks camera support. You should also know that Simulator executes x86 native code, while real iOS devices are running on ARM. This difference will not allow you to understand the actual performance of your application, which is usually important. You can use a simple device such as iPod Touch, which is quite cheap and may be useful not only for development! But of course, we recommend you to find one of the latest iOS devices; currently these are iPhone 5 and iPad 4. The iOS version should be 6.0 or higher.

When you have all the hardware, you'll need to install Xcode—the cornerstone of all Mac-centric development. You will need Version 4.6.3 or higher. We recommend installing it together with the command-line tools, so you'll have access to compiler and some other tools from the Terminal.

We're almost ready to start development, and you can actually proceed to the following How to do it... section if you're going to use Simulator. But if you're going to use a real device now or later, there is one more step. In order to run your application on a real device, you have to become a registered Apple developer (which is free), and you will also need to subscribe for the iOS Developer Program, which will cost you $99 per year. This may look too high for the opportunity to play with your little applications, but this is how Apple verifies that you're serious, and that you will do your best to create a decent app. Also, it gives you access to all beta software from Apple, related to iOS, which is very important from a developer's perspective. Registration procedure is described at https://developer.apple.com/register/index.action, and the page about the "iOS Developer Program" is at https://developer.apple.com/programs/ios/. Finally, you need to register your device according to this instruction found at http://bit.ly/3848_DeviceProvisioning.

Source code for this recipe is available in the Recipe01_HelloWorld folder in the code bundle that accompanies this book.

So, now we're ready to create our first "Hello World" application, and it's going to be in Objective-C. The following are the steps required to achieve our goal:

Let's implement the described steps:

The NSLog function, which we added first, is intended for logging simple text messages, similar to the printf function in the C language. You can also see that our string was preceded with the @ character, which is used for implicit conversion to the NSString object. Just like printf, NSLog allows you to print values of multiple variables with proper formatting, so this function is quite helpful during debugging and profiling of your application.

The next code block shows how one can create a simple message window with some notification. We are not going to use UIAlertView in later recipes, but this is a good occasion to get familiar with creating objects and calling their methods in Objective-C. The first line shows how a new object may be created. Here, the alloc method is called that was inherited by UIAlertView from its parent NSObject class. Next, we're calling the initWithTitle method, passing necessary arguments to it. This method returns a newly initialized alert window. Finally, we call the show method to display the window with our message.

Please note that we've reformatted the code for readability, and we've even created one surplus temporary object. It was possible to call the initWithTitle method of a newly allocated object, so in most cases, you will likely meet the initialization code as shown in the following code snippet.

UIAlertView * alert = [[UIAlertView alloc] initWithTitle:@"Hey there!" message:@"Welcome to OpenCV on iOS development community" delegate:nil cancelButtonTitle:@"Continue" otherButtonTitles:nil];

That's all for this recipe, but if you feel that you need some more introductory information on iOS development in general, we'll provide you with some pointers. Later, we will focus mostly on OpenCV-related aspects of programming, so if you want to better understand Apple's tools and frameworks, you should spend some time studying other resources.

Source code for this recipe is available in the Recipe02_DisplayingImage folder in the code bundle that accompanies this book. You can also take your own image with the preferred 320 x 480 resolution. Or, you can use the provided lena.png image, based on the famous picture among computer vision engineers (http://lenna.org). You can use iOS Simulator to work on this recipe.

The following are the steps required to display an image:

Let's implement the described steps:

In this recipe we have implemented our first GUI on iOS. We'll now discuss some basic concepts related to GUI development. The most important idea is using Model-View-Controller design pattern, which separates visual representation, user interaction logic, and the core logic of the application. There are three parts in this pattern:

Usually, applications have several Views with some rules to switch between them. Also, simple programs usually contain only two parts of the pattern: View and Controller, because logic is very simple, and developers do not create a separate entity for the Model.

A View is created as a storyboard element. The file with the *.storyboard extension allows you to describe an interface of your application with all internal elements. Xcode contains a special graphical tool to add visual controls and change their parameters. So, all that you need is to fill your View with the needed GUI components using drag-and-drop.

When you create a new project, Xcode adds two storyboards for different device families (MainStoryboard_iPhone.storyboard and MainStoryboard_iPad.storyboard). Of course, you can use a single storyboard for all devices. For this purpose, you should change value of the Main Storyboard property in Deployment Settings of the project. But tablets and smartphones differ much in screen resolutions, so it is highly recommended to create separate Views with different layouts for both families.

For each View, you should normally have a Controller. For every new project, Xcode creates a ViewController class by default (ViewController.h and ViewController.m files). In our example, we first add the IBOutlet property to the interface declaration of our View. IBOutlet is a special macro to denote a variable that can be attached to some visual component on the View. IBOutlet resolves to nothing, but it makes clear to Xcode that such variables can be linked with UI elements.

In our implementation, we use the @property keyword. By default, if we add some variable to the Controller's interface (as well as to any other interface), it will be private, so we can't access it out of the class. If we want to do it, we can use the @property keyword. It is somewhat added as an instance variable, but it requires you to implement getter and setter methods. In our example, we do it by calling another special @synthesize keyword. It automatically generates getter and setter methods for your variable.

In this recipe, we add some code to the viewDidLoad method of the ViewController class. This method is a good place to show our image, because it is called after the ViewController has been loaded. You may have noticed that this method already had the following line:

[super viewDidLoad]; 

It is just a call of the viewDidLoad method implemented in the superclass. Here we use UIImage object to load an image from the file. UIImage is a high-level class to store and display image data. It is similar to cv::Mat class as an image container, but can't be used for mathematical computations.

To display the image on the View, we just need to assign the variable with the loaded image to the imageView.image property.

You have the possibility to implement getters and setters manually:

-(UIImageView*)imageView
{
   return imageView1;
}

-(void)setImageView:(UIImageView*)newImageView
{
  if (newImageView != imageView1)
  {
   imageView1 = newImageView; 
  }
}

First you should download the OpenCV framework for iOS from the official website at http://opencv.org. In this book, we will use Version 2.4.6. You can use the iOS Simulator to work on this recipe. Source code for this recipe can be found in the Recipe03_LinkingOpenCV folder in the code bundle that accompanies this book.

The following are the main steps to accomplish the task:

Let's implement the described steps:

Now run your application and check whether the application finds edges on the image correctly.

Frameworks are intended to simplify the process of handling dependencies. They encapsulate header and binary files, so the Xcode sees them, and you don't need to add all the paths manually. Simply speaking, the iOS framework is just a specially structured folder containing include files and static libraries for different architectures (for example, armv7, armv7s, and x86). But Xcode knows where to search for proper binaries for each build configuration, so this approach is the simplest way to link external library on the iOS. All dependencies are handled automatically and added to the final application package.

Usually, iOS applications are written in Objective-C language. Header files have a *.h extension and source files have *.m. Objective-C is a superset of C, so you can easily mix these languages in one file. But OpenCV is primarily written in C++, so we need to use C++ in the iOS project, and we need to enable support of Objective-C++. That's why we have set the language property to Objective-C++. Source files in Objective-C++ language usually have the *.mm extension.

To include OpenCV header files, we use the #import directive. It is very similar to #include in C++, while there is one distinction. It automatically adds guards for the included file, while in C++ we usually add them manually:

#ifndef __SAMPLE_H__
#define __SAMPLE_H__
…
#endif

In the code of the example, we just convert the loaded image from a UIImage object to cv::Mat by calling the UIImageToMat function. Please be careful with this function, because it entails a memory copy, so frequent calls to this function will negatively affect your application's performance.

After converting images, we do some simple image processing with OpenCV. First, we convert our image to the single-channel one. After that, we use the GaussianBlur filter to remove small details. Then we use the Canny method to detect edges in the image. To visualize results, we create a white image and change the color of the pixels that lie on detected edges. The resulting cv::Mat object is converted back to UIImage and displayed on the screen.

The following is additional advice.

// Create file handle
NSFileHandle* handle =
    [NSFileHandle fileHandleForReadingAtPath:filePath];
// Read content of the file
NSData* data = [handle readDataToEndOfFile];
// Decode image from the data buffer
cvImage = cv::imdecode(cv::Mat(1, [data length], CV_8UC1,
                       (void*)data.bytes),
                       CV_LOAD_IMAGE_UNCHANGED);

Source code for this recipe can be found in the Recipe04_DetectingFaces folder in the code bundle that accompanies this book. For this recipe, you will need to download the XMLfile from the OpenCV sources at http://bit.ly/3848_FaceCascade. Alternatively, you can find the file in the resources for this book. You can use the iOS Simulator to work on this recipe.

The following are the basic steps needed to accomplish the task:

Let's implement the described steps:

The first steps of this example are similar to ones from previous recipes. You should create an Xcode project, add the OpenCV framework, add a UIImageView component to the storyboard, and load an input image from the project resources. We just add some more complex OpenCV functionality to detect faces.

In the next recipes, we will discuss how to detect faces in a live video stream, but right now, let's try to do it for a static image. For this task, we use the cv::CascadeClassifier class. The Haar-based OpenCV face detector was initially proposed by Paul Viola and later extended by Rainer Lienhart. It is based on Haar features and allows finding some specific objects. This method is the de facto standard for face detection tasks. The input XML file contains parameters of such classifiers trained to detect frontal faces.

To load parameters, we need to convert the NSString object to std::string. In order to do it, we use the UTF8String method that returns a null-terminated UTF-8 representation of the NSString object.

After that, we can find faces on our image with the help of the detectMultiScale method of the cv::CascadeClassifier class.

OpenCV function. This function receives the following parameters to configure the detection stage:

Detailed description of the function arguments you can be found in the OpenCV documentation at http://bit.ly/3848_DetectMultiScale.

This function is parallelized with Grand Central Dispatch, so it will work faster on multi-core devices.

Each detected rectangle is added to the resulting image with the cv::rectangle function.

Now you can try to replace lena.png with your family photo or some other image with faces.

Object detection is a wide and deep subject, and we only scratched its surface in this recipe. The following will give you some pointers if you want to know more.

The source code for this recipe is in the Recipe05_PrintingPostcard folder in the code bundle that accompanies this book, where you may find implementation of the PostcardPrinter class and images that are going to be used in our application. You can use the iOS Simulator to work on this recipe.

The following is how we can implement a postcard printing application:

Let's implement the described steps:

You can now build and run your application to see the result.

You can see that the PostcardPrinter class is an ordinary C++ class, and it actually could be used in any desktop application. We won't discuss its implementation in details, as it is not iOS-specific and is implemented using simple OpenCV functions. We will only mention that the crumpling effect is implemented by changing intensity values of images, and this is done in HSV color space. We'll first calculate the value of relief using texture, and then subtract it from the intensity plane of the image (value channel in HSV).

In viewDidLoad, we first load the images. You can note that the params.texture member is converted to RGB color space to comply with the input format of PostcardPrinter. But the params.text is loaded with the alpha channel, which is later used to avoid font aliasing. The last Boolean argument in the UIImageToMat function indicates that we need this image to be converted with alpha:

UIImageToMat(image, params.text, true);

Also note that C++ code can be seamlessly called from the Objective-C code. That's why in viewDidLoad, we simply create a PostcardPrinter object and then call its methods.

Finally, note how time measurements are added. OpenCV's getTickCount and getTickFrequency functions are used. The following line of code can be used to calculate time in seconds, so we multiply it with 1000 to get the printing time in milliseconds.

float(timeEnd - timeStart) / cv::getTickFrequency();

Depending on your device, the time will vary, but it shouldn't exceed half a second, which is good enough for our use case. Later, we will not use the getTickCount function directly, rather we'll use helper macros:

  #define TS(name) int64 t_##name = cv::getTickCount()
  #define TE(name) printf("TIMER_" #name ": %.2fms\n", \
    1000.*((cv::getTickCount() - t_##name) / cv::getTickFrequency()))

It actually does the same measurement, but it greatly improves readability of the code under inspection. It allows us to measure the working time in a simple manner as shown in the following code:

    // Print postcard
    cv::Mat postcard;
    TS(PostcardPrinting);
    postcardPrinter.print(postcard);
    TE(PostcardPrinting);

Our PostcardPrinter class is quite simple, but you can use it to start a full-featured application. You can add more textures, effects, and fonts, so users have more freedom while designing their postcards. In the next recipe, we apply a cool vintage effect to the photo, so it looks like a real poster from the XIX century. Its OpenCV implementation is quite simple, so we leave it for self-study.

Source code for this recipe can be found in the Recipe06_WorkingWithGallery folder in the code bundle that accompanies this book. You can use the iOS Simulator to work on this recipe, but you will also need an image with a face in your Gallery. You can open the Safari browser on the Simulator, copy lena.png from your PC using the drag-and-drop method, then save it using the long mouse click.

The first steps are the same as in the previous recipes. We should create an Xcode project, reference the OpenCV framework, add the UIImageView component, and copy files with the postcard printing code.

We are going to add the possibility to print a postcard with an image from Gallery (the image should have face in it) and save the resulting image back to Gallery. The following are the steps required to do it:

Let's implement the described steps:

The remaining functions (for example, viewDidLoad) were not changed much, so we'll not explain them in detail.

For loading images from Gallery, we have to use the UIImagePickerController class. It provides user interfaces for choosing images and videos from your device. In order to use it, we should implement several protocols in our ViewController class, so it becomes a delegate that conforms to these protocols. This way, it follows the so-called delegation mechanism that is widely used in iOS. It allows you to avoid inheriting from base classes, and instead, the delegation requires implementing a protocol with some particular methods. In our recipe, we will use three delegates for taking images from Gallery: UIImagePickerControllerDelegate, UINavigationControllerDelegate, and UIPopoverControllerDelegate. To implement the first protocol, we should add the imagePickerControllerDidCancel method that will be called if the user presses the Cancel button before choosing an image in Gallery. In our recipe, we are just closing the window with the user's photo in this case. We should also implement the didFinishPickingMediaWithInfo method that describes the application behavior if the user selects an image. In our case, we call the printPostcard method for the selected image and store the result to the image global variable.

As you may have noticed, all these functions have a conditional statement that checks whether we use iPad or iPhone. It follows the UI guidelines for iOS. On iPhone and iPod devices, it is common to use a full screen window to show photos from Gallery, but on iPad, we have to use pop-up windows. So, to close the window, we should use two different implementations for corresponding classes of devices.

In the action of the Load button, we should create a UIImagePickerController object and set the delegate property of this object to our ViewController class. It allows us to inform our Controller about the changes and invoke the created implementation of the protocol. In the case of iPhone/iPod, we should just present the Controller with the presentViewController method. Implementation for tablets is a bit complicated; we should create a UIPopoverController object and initialize it with the previously created UIImagePickerController object. In order to do it, we will initialize the self.popoverController field that is already contained in our class, because we're using delegation from UIPopoverControllerDelegate.

It is our first recipe, where we were working with buttons. Here we deal with another important Cocoa design pattern called target-action. Actions are messages (the action) that are sent to the Controller (the target) on corresponding button-clicks. In order to process button-clicks, one should catch the corresponding events. For that purpose, you should use the IBAction keyword. IBAction is a special macro that resolves to void, but it denotes a method that can be linked with UI components.

If you want to know more about the delegation and actions, we recommend you to read the Cocoa's Communicating with Objects guide at http://bit.ly/3848_CommunicatingWithObjects.

All the application logic is now in place, but we'll add some features to make our application more user-friendly.

- (NSInteger)supportedInterfaceOrientations
{
    // Only portrait orientation
    return UIInterfaceOrientationMaskPortrait;
}

[saveButton setEnabled:NO];

The source code for this recipe can be found in the Recipe07_ApplyingRetroEffect folder in the code bundle that accompanies this book. You can use the iOS Simulator to work on this recipe.

This recipe heavily relies on the previous one, as we're going to implement the same workflow: loading images from Gallery, processing them with OpenCV, and displaying them on the screen.

The following are the steps required to apply our filter to an image from Gallery:

Let's implement the described steps:

The remaining Objective-C code is based on the previous recipe, so it is not shown here.

The only new information in this recipe is the implementation of the RetroFilter class. It uses popular OpenCV functions, and we will explain only its most interesting part—the applyToPhoto method.

This method applies a sequence of processing steps that help us to achieve a "retro" effect. The key idea is to convert an image to a monochrome color space, do all the processing in it, and eventually convert it back to RGB with the sepia effect.

Both scratches and borders are rendered with a color that depends on a mean color of the image. To avoid costly analysis of the whole image, we only look into middle row of the image:

Scalar meanColor = mean(luminance.row(luminance.rows / 2));

You can also see that we are using the cv::RNG class (initialized with rng_(time(0))) to choose a region on the image with scratches randomly. This allows us to get different patterns of scratches for different images.

Finally, we assemble back the channels of our image. We add a value of 20 to the luminance plane, so the contrast is artificially decreased. After that, we use the OpenCV merge function to pack color planes into the single image, then convert it to the RGB color space with the help of the cvtColor function.

You can try to use your own images with scratches and borders. As before, we recommend you to use GIMP software to edit images. But please note that both scratches.png and fuzzyBorder.png should be one-channel images, because they are used as a mask and alpha channel correspondingly.

For this recipe, you will need a real iOS device, because we're going to take photos. The source code can be found in the Recipe08_TakingPhotosFromCamera folder in the code bundle that accompanies this book.

The following are the steps required to apply our filter to a photo, taken with camera app:

Let's implement the described steps:

In order to work with a camera on an iOS device using OpenCV classes, you need to initialize the CvPhotoCamera object first and set its parameters. This is done in the viewDidLoad method that is called once when the View is loaded onscreen. In the initialization code, we should specify what GUI component will be used to preview the camera capture. In our case, we'll use UIImageView as we did before.

Our main UIImageView component will be used to show the video preview from the camera and help users to take a good photo. Because our app also needs to display the final result on the screen, we create another UIImageView to display the processed image. In order to do it, we can create the second component right from the code:

resultView = [[UIImageView alloc]
                  initWithFrame:imageView.bounds];    
UIImage* result = [self applyEffect:image];   
[resultView setImage:result];
[self.view addSubview:resultView]; 

In this code, we create the UIImageView component with the same size as that of manually added imageView property. After that, we use the addSubview method of the main View to add newly created components to our GUI. If we want see the camera preview results again, we should use the same method for the imageView property:

[self.view addSubview:imageView];

There are three important parameters for camera: defaultAVCaptureDevicePosition, defaultAVCaptureSessionPreset, and defaultAVCaptureVideoOrientation. The first one is designed to choose between front and back cameras of the device. The second one is used to set the image resolution. The third parameter allows you to specify the device orientation during the capturing process.

There are many possible values for the resolution; some of them are as follows:

For capturing static, high-resolution images, we recommend using the value of AVCaptureSessionPresetPhoto. The resulting resolution depends on your device, but it will be the largest possible resolution.

In order to start the capture process, we should call the start method of the camera object. In our sample, we'll do it in the button's action. After clicking on the button, the user will see the camera image on the screen and will be able to click on the Take photo button that calls the takePicture method.

The CvPhotoCameraDelegate camera protocol contains only one important method—capturedImage. It is executed when somebody calls the takePicture function and allows you to get the current frame as the function argument.

If you want to stop the camera capturing process, you should call the stop method.

If you want to start capturing at the time the application is launched, you have to call the start method inside viewDidAppear:

- (void)viewDidAppear:(BOOL)animated
{
    [photoCamera start];
}

The source code for this recipe is available in the Recipe09_CreatingStaticLibrary and CvEffects folders in the code bundle that accompanies this book. You can use the iOS Simulator to work on this recipe.

This recipe is actually a refactoring of our Recipe05_PrintingPostcard project. We will split it into two, one will be a core computer vision library written in C++, and the second will be an iOS application, written in Objective-C.

The following is the high-level description of the required steps:

Let's implement the described steps:

The setup is complete; now you can build and run the application.

The static library project on iOS doesn't differ much from the static libraries for other platforms, so we'll not dig deeper into this subject. As you can see, such project type allows you not only to keep the source files, but also to keep the images and other resources. This is a good opportunity to reduce duplication in your code base.

For a more detailed introduction into static libraries in iOS, you can refer to the official documentation at http://bit.ly/3848_iOSStaticLibraries.

A static library is a good way to reuse your code between projects, but there are some additional opportunities for developers. Let's discuss them.

The source code can be found in the Recipe10_CapturingVideo folder in the code bundle that accompanies this book. For this recipe, you can't use Simulator, as it doesn't support camera.

The high-quality camera, in the latest iOS devices, is one of important factors of the popularity of these devices. The ability to capture and encode H.264 high-definition video with hardware acceleration was accepted with great enthusiasm by users and developers.

Most of the functions related to communicating with camera are included in the AVFoundation framework. This framework contains a lot of simple and easy-to-use classes for taking photos and videos. But setting up a camera, retrieving frames, displaying them, and handling rotations, take a lot of code. So, in this recipe, we will use the CvVideoCamera class from OpenCV, which encapsulates the functionality of the AVFoundation framework.

The following are the steps required to capture video on iOS:

Let's implement the described steps:

As we mentioned earlier, the iOS part of the OpenCV library has two classes for working with camera: CvPhotoCamera and CvVideoCamera. The difference between the two classes is rather conventional. The first one was designed to only capture static images and you can process images only after capturing them (offline mode). The other class provides more opportunities. It can capture video, process it on the fly, and save the processed stream as an H.264 video file. Those classes have a quite similar interface and are inherited from the common CvAbstractCamera ancestor.

The CvVideoCamera class is easy to use. You can leave the default values for resolution, frames-per-second (FPS), and so on, or customize them when needed. The parameters are the same as the ones in the CvPhotoCamera class; however, there is one new parameter called defaultFPS. Usually, this value is chosen between 20 and 30; 30 being standard for video.

Previously, we recommended using AVCaptureSessionPresetPhoto as a resolution parameter of the CvPhotoCamera class. In case of video capturing, the better way is to choose a smaller resolution. In order to do so, you can use one of the fixed resolutions (for example, AVCaptureSessionPreset640x480, AVCaptureSessionPreset1280x720, and so on) or one of the relative ones (AVCaptureSessionPresetHigh, AVCaptureSessionPresetMedium, and AVCaptureSessionPresetLow). The resulting resolution in the latter case will depend on the respective device and camera. Some of the values are listed in the following table:

To work with camera on an iOS device using the OpenCV class, you should first initialize the CvVideoCamera object and set its parameters; you can do it in the viewDidLoad method.

In order to start the capturing process, we should call the start method of the camera object. In our sample, we'll do it in the button's actions (callback functions). After pressing the button, the user will see the camera preview on the screen. In order to stop capturing, you should call the stop method. You should also implement the processImage method that allows you to process camera images on the fly; this method will be called for each frame. Its input parameter is already converted to cv::Mat that simplifies calling the OpenCV functions.

It is also recommended to stop the camera when the application is closing. Add the following code to guarantee that the camera stops in case the user doesn't click on the Stop capture button:

- (void)viewDidDisappear:(BOOL)animated
{
    [super viewDidDisappear:animated];
    if (isCapturing) {
        [videoCamera stop];
    }
}

CvVideoCamera simply wraps AVFoundation functions. So, if you need more control on the camera, you should use this framework directly. The other way is to add OpenCV classes for working with the camera to your project directly. For that purpose, you should copy cap_ios_abstract_camera.mm, cap_ios_photo_camera.mm, cap_ios_video_camera.mm, and cap_ios.h from the highgui module and modify the included files. You will need to rename the classes to avoid conflict with the classes of OpenCV.

Real-time video processing on mobile devices is often a computationally intensive task, so it is recommended to use dedicated frameworks, such as Accelerate and CoreImage. Such frameworks are highly optimized and accelerated with special hardware, so you can expect decent processing time and significant power savings.

We will use the Recipe10_CapturingVideo project as a starting point, trying to add more control over the iOS camera. The source code can be found in the Recipe11_AdvancedCameraControl folder in the code bundle that accompanies this book. For this recipe you can't use Simulator, as it doesn't support camera.

The following are the required steps:

Let's implement the described steps:

First, let's investigate the focus changing on iOS devices. The iOS camera API supports three modes for camera focus:

CvVideoCamera uses the AVCaptureFocusModeContinuousAutoFocus mode by default, so the focus may change with the scene. This can make the process of camera calibration much more difficult. The best way, in this case, is to set the focus to some special value (for example, infinity), but unfortunately, the iOS API doesn't contain all functions needed by computer vision specialists. There is no way to programmatically set the camera focus of an iOS device to infinity or any other predefined value. So we can only lock the current focus value. For that purpose, the CvVideoCamera class provides the lockFocus method. It changes the focus mode to the AVCaptureFocusModeLocked value. In order to unlock it again, you should use the unlockFocus function.

You can also control the exposure and white balance in the same way using lockExposure and lockBalance functions.

To control image rotation in camera, you can change the rotateVideo property. The default value of this variable is NO.

In this mode, the camera image will not be rotated with the device rotation. If you change this value, the image will be rotated every time a device changes its orientation by 90 degrees.

Each newly added button in this project allows you to switch between the two modes. To indicate mode changing, we'll change the button's text. For that purpose, you can use the setTitle method.

AVFoundation contains a lot of useful functions for advanced control of the iOS camera. For example, you can get exposure time for each frame. If you need such fine-grain control, you should work with AVFoundation directly.

We will use the Recipe10_CapturingVideo project as a starting point, trying to apply previously implemented RetroFilter to the video stream. We also suppose that the RetroFilter class, and its resources were added to the CvEffects static library project. Source code can be found in the Recipe12_ProcessingVideo folder in the code bundle that accompanies this book. For this recipe, you can't use Simulator, as it doesn't support working with camera.

The following are the required steps:

Let's implement the described steps:

In the previous cases, we always created the filter object right before using it. In the case of live video, we cannot do it, because the performance issues come out on top. So we'll initialize the RetroFilter object only once. For this purpose, we have to add a smart pointer, which points to the filter object, to the Controller interface and initialize it after starting the video capturing process. We can't do it in the viewDidLoad method, because we should know the camera resolution from before.

To calculate FPS, we have to add the prevTime field property. We will measure the time between processImage calls with this variable. At the time of the first call to this method, we'll initialize this property with the current time. During the next call, we will be able to measure the working time of the filter function, plus the time needed to get the camera image as a difference between current time and value of the prevTime variable. After that, we can convert it to seconds and calculate the resulting FPS value. In order to display the number on the screen, we'll use the cv::putText function.

Even on the latest iOS devices (iPad 4 and iPhone 5) our filter shows good FPS (~30) only on low resolutions, for example 352 x 288. In the next recipes, we'll consider a few ways to optimize the OpenCV applications with iOS- and ARM-specific techniques.

We will use the Recipe12_CapturingVideo project as a starting point, trying to add the possibility to save processed camera images as video. The source code can be found in the Recipe13_SavingVideo folder in the code bundle that accompanies this book. For this recipe, you can't use Simulator, as it doesn't support camera.

One of the many features supported by the latest iOS devices is 1080p HD video recording. It is incredible, because it is the largest resolution for the majority of modern TVs. Also, the modern iOS devices support hardware encoding with the H.264 codec and QuickTime container (.mov).

The following are the required steps:

Let's implement the described steps:

At a high level, the main method of the CvVideoCamera class (that captures frames and processes them) looks like the following:

while (flag)
{
    if (self.delegate)
    {
        //Get current frame
        
        //Call process function
        
        if (self.recordVideo == YES) {
            //Add the image to video file
        }
    }
}

In other words, the processImage method will be called for each frame after calling the start method. And each processed frame is saved to the resulting video file if the value of the recordVideo variable is YES.

When a user clicks on the Stop capture button, the stopCaptureButtonPressed method is called. In this function, we have to call the stop method initially. After that, the resulting video file becomes available in tmp folder of the current application. You can copy it from device with some software, such as iFunBox. You can get the path to this file through the videoCamera.videoFileURL property. In order to copy it to Gallery, we should call the UISaveVideoAtPathToSavedPhotosAlbum function using the path to the generated file. After that, we can show an alert window with a notification message about video saving as done in the Capturing a video from camera (Simple) recipe.

We will use the Recipe12_ProcessingVideo project as a starting point, trying to minimize the processing time. The source code is available in the Recipe14_OptimizingWithNEON folder in the code bundle that accompanies this book. For this recipe, you can't use Simulator, as NEON instructions are not supported on it and they are ARM-specific, while Simulator is x86.

The following is how we will optimize our video processing application:

Let's implement the described steps:

Nowadays, SIMD instructions are available on many architectures, from desktop CPU to embedded DSP. ARM processors provide a rich set of instructions, called NEON; they are available on all iOS devices starting from iPhone 3GS.

To start writing NEON code, you have to add the following declaration to your file:

#if defined(__ARM_NEON__)
  #include <arm_neon.h>
#endif

Now you can use all the types and functions declared there. Please note, that we're going to use the so-called intrinsics—functions in C that serve as a wrapper over NEON assembler instructions. In fact, you can write your code in pure assembler, but it will worsen the readability, although there is a small performance gain, it usually isn't worth it.

Let's consider how the alphaBlendC1_optimized function works. This function should use the following formula to calculate the resulting pixel's value:

dst(x, y) = [alpha(x, y) * src(x, y) + (255.0 - alpha(x, y)) * dst(x, y)] / 255.0;

The NEON code does exactly that, except the very last division, which is approximated by bit-shifting 8 positions to the right (vshrn_n_u16 function). This means that we divide by 256, instead of 255, and the result of the vectorized function may differ from the original implementation. But we can tolerate that, as we're working on a visual effect, and the possible difference is negligibly small. But please note that such approximations may be unacceptable in a numerical pipeline.

You can also see that we process 8 pixels simultaneously. Our alphaBlendC1_optimized function heavily relies on the exact format of input matrices (that is, is one channel, is continuous, and the number of columns is a multiple of 8), but it can be easily generalized for other situations.

The multiply function performs simple multiplication with a floating-point coefficient. But we need to do a sequence of conversions to perform the multiplication. But still, because we process 8 pixels simultaneously, the speedup is impressive.

Performance optimization with NEON is a deep and wide subject. Most image processing functions could be optimized for 3x speedup, without affecting accuracy. You can even get more if you apply some approximations. In the following sections, we provide some pointers for further study.

    uchar src = 111;
    float multiplier = 0.76934;
    uchar dst = 0;

    dst = uchar(src * multiplier);
    printf("dst floating-point = %d\n", dst);

    uchar fpMultiplier = uchar((multiplier * 128.f) + 0.5f);
    dst = (src * fpMultipiler) >> 7; // 128 = 2^7
    printf("dst fixed-point = %d\n", dst);

    dst floating-point = 85
    dst fixed-point = 84

The source code for this recipe is available in the Recipe15_DetectingFacialFeatures folder in the code bundle that accompanies this book. You can't use Simulator, as we're going to use camera in this recipe.

The following are the steps required to implement the application for this recipe:

Let's implement the described steps:

Let's consider how the detectAndAnimateFaces method works. You can see that the processing time of every step is measured, as the overall processing is quite expensive.

We are already familiar with detecting objects (and faces in particular) using OpenCV's CascadeClassifier class. You can see that we use a different cascade in this example, which is based on LBP-features (Local Binary Patterns). This cascade works several times faster than the Haar-based cascade and the quality doesn't differ much. And this performance difference is important, because we're going to process live video.

When the detection is completed, we sort the vector of detected faces by their size using the FaceSizeComparer function. The for loop is used to detect facial features within every face. We decided to detect eyes in every even face, and mouth in every odd face.

We use a couple of tricks to improve the quality and minimize the detection time. First of all, we limit the search area, so that eyes are detected on the upper half of the face rectangle, and the mouth in the lower half. This not only improves the performance, but also allows avoiding false detections. Secondly, we search only for the largest object using the CV_HAAR_FIND_BIGGEST_OBJECT flag. It stops the detection when the first object is found, so we don't waste our time searching for another pair of eyes or mouth in the same face rectangle. It is obvious that even if we find something, this should be a false detection. Finally, we control the minimal facial feature size. The following are empirically found minimal relative sizes for eyes and mouth:

Size(faces[i].width * 0.55, faces[i].height * 0.13) //eyes
Size(faces[i].width * 0.20, faces[i].height * 0.13) //mouth

Finally, we put some animation over the detected facial feature, using the alpha blending function from the previous recipes.

This sample presents the very basic approach to facial feature detection. It can be significantly improved in both quality and speed. Let's consider some opportunities.

All the source code changes will be localized in the CvEffects library, so you can use the same Recipe15_DetectingFacialFeatures project. Again, you can't use Simulator, as we're going to use the camera.

In the previous recipe, we have profiled the FaceAnimator class and slightly improved the facial feature detection time by tuning the parameters. But the very first preprocessing step was still quite expensive, and we're going to optimize it using the Accelerate framework. In fact, we could work on a custom NEON optimization as before, but Accelerate could be a good time saver, as it provides a wide set of optimized functions for image processing. We will replace cv::cvtColor and cv::equalizeHist with calls to Accelerate functions. Histogram equalization helps the detection algorithm to better tolerate illumination changes.

The following are the steps required to accomplish the task:

Let's implement the described steps:

First of all, you should see that Accelerate uses the vImage_Buffer structure as an image container. This structure is quite simple, but more importantly, it can be created on top of the existing OpenCV matrix. We don't have to copy or convert data, and that allows to seamlessly interleave calls to OpenCV and Accelerate, without any performance penalty. The following is how we initialize vImage_Buffer using cv::Mat data:

    vImagePixelCount rows = static_cast<vImagePixelCount>(src.rows);
    vImagePixelCount cols = static_cast<vImagePixelCount>(src.cols);
    vImage_Buffer _src = { src.data, rows, cols, src.step };

Unfortunately, Accelerate doesn't provide color space conversions, and we have to use the generic vImageMatrixMultiply_ARGB8888 transformation function. So, cvtColor_Accelerate is implemented in two steps, we first convert the RGBA input matrix to another four-channel matrix, first channel of which is the required grayscale image. Then we split the resulting matrix into four planes, and later use only the first one.

It should be noted that vImageMatrixMultiply_ARGB8888 actually uses fixed-point arithmetic, and the numbers in the matrix variable are RGBA to Gray conversion coefficients, multiplied by 256. That's why we use 256 to initialize the divisor:

Y = 0.299R + 0.587G + 0.114B ≈ (77R + 150G + 29B) / 256

After the conversion, we use vImageConvert_ARGB8888toPlanar8 to get the first channel with the image intensity data.

Implementation of the equalizeHist_Accelerate is much more straightforward. We simply call the vImageEqualization_Planar8 function, and use its result directly.

As a final note, the Accelerate framework (in contrast to OpenCV) wants the user to allocate all the input and output buffers manually, and of course to deallocate this memory later. That's why we call the Mat::create method for three image buffers in the PreprocessToGray_optimized method. You shouldn't be afraid of slow memory reallocations on every frame, as OpenCV doesn't recreate matrices if they are already in the desired format.

We used only three functions from the Accelerate framework, but there are many more of them. Please refer to the official documentation if you want to know more: http://bit.ly/3848_Accelerate. You will see that many primitives from OpenCV's core and imgproc modules can be found there. Despite the fact that Accelerate's syntax is somewhat noisy, the use of this framework could be a cheaper solution, than to manually optimize every function with NEON. You should also note that Accelerate not only tries to exploit CPU (with NEON extensions), but also Digital Signal Processor (DSP), so it could provide a better speedup than a manually vectorized code.

There is no source code for this recipe, as we're going to build OpenCV. You will need a Git command-line client, CMake (Version 2.8.11 or higher), and Python 2.7 installed. Usually, Python is already installed on Mac OS, but the CMake tool needs to be downloaded from http://cmake.org. And, you don't need an iOS device, because the compilation is done on a host computer (so-called cross-compilation).

The following are the steps required to get your custom OpenCV build:

Let's implement the described steps:

As we mentioned before, iOS frameworks are a better way to distribute your static libraries. In its core, they are simple libraries and headers, but they may contain binary code for several architectures (such as armv7, armv7s, and i386 in our example). That makes them more convenient to link from Xcode projects, because you need not think about linker configuration, rather, you can simply add a reference to the framework.

So, the only interesting moment in this recipe is how the build_framework.py script constructs the OpenCV framework. You can actually study its source code to get complete understanding. For every architecture (old and new iOS devices, plus Simulator), it generates an Xcode project using CMake and executes Xcode in order to build it. When all three configurations are built, script forms the ~/<my_working_directory>/build_ios/opencv2.framework directory properly, so it becomes a valid iOS framework.

When the script finishes its work, you can use the built framework as a normal OpenCV distribution.

Despite the fact that it is quite easy to create your custom version of OpenCV, we encourage you to use the official one. The library is always in active development; new versions are rolled out regularly, so if you don't want to waste too much time merging your changes, better to stick to the official distribution.

All new source code should be developed outside of the library itself, as we did with the CvEffects project. And, if your development grows into something stable and useful, you can always contribute your code as a new OpenCV module, or as an extension to the existing one.

In case you've found a bug, you can live with your custom build for some time. But you should submit a GitHub pull request with the fix, so it is integrated into the development branches as soon as possible. The same rule applies to performance optimizations. If your code is faster, but still generic enough (you have tested it on multiple platforms and with different parameters), you can submit a pull request. This way we will have an even more stable and efficient library!

More information on the contribution process is available on the official website at http://opencv.org/contribute.html.