Android Application Programming with OpenCV

4.7 (3 reviews total)
By Joseph Howse
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies

About this book

Take a smartphone from your pocket, and within a few seconds, you can snap a photo, manipulate it, and share it with the world. You have just achieved mass production of image data. With a computer vision library such as OpenCV, you can analyze and transform copious amounts of image data in real time on a mobile device. The upshot to this is that you, as developers, can provide mobile users with many new kinds of images, constantly highlighting certain visual features that are of artistic or practical interest. Android is a convenient platform for such experiments because it uses a high-level language (Java), it provides standardized interfaces for sharing image data between applications, and it is mostly open source, so everyone can study its implementation.

Android Application Programming with OpenCV is a practical, hands-on guide that covers the fundamental tasks of computer vision—capturing, filtering, and analyzing images-with step-by-step instructions for writing both an application and reusable library classes.

Android Application Programming with OpenCV looks at OpenCV's Java bindings for Android and dispels mysteries such as which version of these bindings to use, how to integrate with standard Android functionality for layout, event handling, and data sharing, and how to integrate with OpenGL for rendering. By following the clear, concise, and modular examples provided in this book, you will develop an application that previews, captures, and shares photos with special effects based on color manipulation, edge detection, image tracking, and 3D rendering.
Beneath the application layer, you will develop a small but extensible library that you can reuse in your future projects. This library will include filters for selectively modifying an image based on edge detection, 2D and 3D image trackers, and adapters to convert the Android system's camera specifications into OpenCV and OpenGL projection matrices. If you want a quick start in computer vision for Android, then this is the book for you.

By the end of Android Application Programming with OpenCV, you will have developed a computer vision application that integrates OpenCV, Android SDK, and OpenGL.

Publication date:
September 2013
Publisher
Packt
Pages
130
ISBN
9781849695206

 

Chapter 1. Setting Up OpenCV

This chapter is a quick guide for setting up a development environment for Android and OpenCV. We will also look at the OpenCV sample applications, documentation, and community.

By the end of this chapter, our development environment will include the following components:

  • Java Development Kit (JDK) 6: It includes tools for Java programming.

  • Cygwin 1.7 or greater (Windows only): It is a compatibility layer that provides Unix-like programming tools on Windows.

  • Android Software Development Kit (Android SDK) r21.0.1 or greater: It includes tools for programming Android apps in Java.

  • Android Native Development Kit (Android NDK) r8d or greater: It includes tools for programming Android apps in C++. Although this book deals with Java programming, OpenCV also includes Android-compatible C++ samples that you may want to compile and run.

  • Eclipse 4.2.1 (Juno) or greater: It is an integrated development environment (IDE).

  • Java Development Tools (JDT): It is an Eclipse plugin for Java programming (already included in most Eclipse distributions).

  • C/C++ Development Tooling (CDT) 8.1.1 or greater: It is an Eclipse plugin for C/C++ programming.

  • Android Development Tools (ADT) 21.0.1 or greater: It is an Eclipse plugin for Android programming.

  • OpenCV4Android 2.4.3.2 or greater: It includes OpenCV's Android version, including Java and C++ libraries.

There are many possible ways to install and configure these components. We will cover several common setup scenarios, but if you are interested in even more options, see OpenCV's official documentation at http://docs.opencv.org/doc/tutorials/introduction/android_binary_package/O4A_SDK.html.

 

System requirements


All of the development tools for Android and OpenCV are cross platform. The following operating systems are supported with almost identical setup procedures:

  • Windows XP, Windows Vista, Windows 7, or Windows 8

  • Mac OS 10.6 (Snow Leopard) or greater

  • Ubuntu 10.10 (Maverick) or greater

  • Many other Unix-like systems (though not specifically covered in this book)

To run the OpenCV samples and, later, our own application, we should have an Android device with the following specifications:

  • Android 2.2 (Froyo) or greater (required)

  • Camera (required); front and rear cameras (recommended)

  • Autofocus (recommended)

  • Google Play Store (recommended)

Android Virtual Devices (AVDs) are not recommended. Some parts of OpenCV rely on low-level camera access and may fail with virtualized cameras.

 

Setting up a development environment


Basically, we have a choice among the following approaches:

  1. Install a prepackaged, preconfigured development environment that contains all the components we need.

  2. Install various components separately and configure them to work together. Within this approach, we may do either of the following:

    • Use a prepackaged, preconfigured version of OpenCV

    • Configure and build OpenCV from source

Let's look at each of these alternatives in detail.

Getting a ready-made development environment – Tegra Android Development Pack (TAPD)

Tegra Android Development Pack (TADP) contains a complete, preconfigured development environment for Android, OpenCV, and some other libraries. TADP builds apps that are optimized for NVIDIA's Tegra processors. Despite being optimized for Tegra, the apps are compatible with other hardware too.

Tip

If you are setting up an Android development environment from scratch, I recommend TADP. It contains recent versions of all our required software and its setup process is simple.

TADP also contains some extras that we do not require for this book. For a complete list of TADP's contents, see the official description at https://developer.nvidia.com/tegra-android-development-pack.

To set up TADP, we just need to download and install it from a secure section of NVIDIA's website. Here are the required steps:

  1. Join the NVIDIA Registered Developer Program at https://developer.nvidia.com/user/register. (It is free.)

  2. Log in at https://developer.nvidia.com/user/login.

  3. Complete your user profile at https://developer.nvidia.com/user/me/profile/rdp_profile.

  4. Apply to join the Tegra Registered Developer Program at https://developer.nvidia.com/rdp/applications/tegra-registered-developer-program. (It is free, too!) Wait for NVIDIA to send you an acceptance email. Normally, you might receive it a few minutes after applying.

  5. Go to https://developer.nvidia.com/tegra-resources and find the download link for TADP's latest version. At the time of writing, the latest version is 2.0r2. There are installers for Windows (32-bit or 64-bit), Mac, and Ubuntu (32-bit or 64-bit). Download and run the appropriate installer.

  6. When the installer presents the Installation Directory step, we can enter any destination, which we will refer to as <tadp>. By default, <tadp> is C:\NVPACK (Windows) or ~/NVPACK (Mac and Ubuntu).

  7. When the installer presents the Installation Options step, we may select any option: Complete, Express, or Custom. Compared to an Express installation, a Complete or Custom installation may include additional versions of Android SDK and binary images of Tegra Android OS, which is NVIDIA's customization of Android. If in doubt, choose Express.

  8. When the installer presents the Proxy Configuration step, we may leave all fields blank unless we are using a proxy server.

  9. After finishing all of the installer's configuration steps, wait for TADP's content to be downloaded and installed.

That's all! Before proceeding, let's just take a note of the locations where TADP has installed certain components. For TADP 2.0r2 (the latest version at the time of writing), the locations are as follows:

  • Android SDK is located at <tadp>/android-sdk-macosx. We will refer to this location as <android_sdk>.

  • Android NDK is located at <tadp>/android-ndk-r8d. We will refer to this location as <android_ndk>.

  • OpenCV4Android is located at <tadp>/OpenCV-2.4.3.2-android-sdk-tadp. We will refer to this location as <opencv>.

  • Eclipse is located at <tadp>/eclipse. We will refer to this location as <eclipse>.

Note

The TADP installer automatically edits the system's PATH to include <android_sdk>/platform-tools and <android_sdk>/tools. Also, it creates an environment variable called NDKROOT, whose value is <android_ndk>.

Now, we can proceed to Building the OpenCV Samples with Eclipse, later in this chapter.

Tip

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. The example code for this book is also available from the author's website at http://nummist.com/opencv/.

Assembling a development environment piece-by-piece

Instead of using TADP as a ready-made solution, we may assemble our own development environment. Broadly, this task has two stages:

  • Set up a general-purpose Android development environment

  • Set up OpenCV for use in this environment

Let's start by looking at the setup steps for a general-purpose Android development environment. We will not delve into very much detail here, because good instructions are available at the given links and, because you, as an Android developer, have probably been through similar steps before.

Tip

If you already have an Android development environment and you just want to add components to it, some of the following steps will not apply to you.

Here are the steps:

  1. If we are using Windows or Linux, we may need to obtain JDK 6 manually. (On Mac, if JDK 6 is not present, the operating system will automatically offer to install it when needed.) The JDK 6 installers or packages are available for Windows and many Linux distributions at http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html. Alternatively, on Linux, check your repository for the JDK packages. Install JDK 6.

  2. Download Eclipse and unzip it to any destination, which we will refer to as <eclipse>. There are many versions from which we may choose. Google provides an Eclipse distribution called Android Developer Tools (ADT) Bundle, which comes with Android SDK and the ADT plugin prepackaged and preconfigured. ADT Bundle is available at http://developer.android.com/sdk/index.html. Many other up-to-date Eclipse distributions are available at http://www.eclipse.org/downloads/. Of these, Eclipse for Mobile Developers is a good and minimalist choice as a foundation for an Android development environment.

  3. If we did not get the ADT Bundle, we now need to set up Android SDK and the ADT plugin for Eclipse. Go to http://developer.android.com/sdk/index.html and get the download named Android SDK Tools. Install or unzip it to any destination, which we will refer to as <android_sdk>. Open Eclipse and install the ADT plugin according to the official instructions at http://developer.android.com/sdk/installing/installing-adt.html. Restart Eclipse. A window, Welcome to Android Development, should appear. Click on Use Existing SDKs, browse to <android_sdk>, and click on Next. Close Eclipse.

  4. If we are using Windows, download and install Cygwin from http://cygwin.com/install.html.

  5. Download Android NDK from http://developer.android.com/tools/sdk/ndk/index.html. Unzip it to any destination, which we will refer to as <android_ndk>.

  6. Edit your system's PATH to include <android_sdk>/platform-tools and <android_sdk>/tools. Also, create an environment variable named NDKROOT with the value as <android_ndk>. (If you are unsure how to edit PATH and other environment variables, see Appendix A: Editing environment variables.)

    Tip

    Editing environment variables on Windows

    The system's Path variable and other environment variables can be edited in the Environment Variables window of Control Panel.

    On Windows Vista/7/8, open the Start menu and launch Control Panel. Now, go to System and Security | System | Advanced system settings. Click on the Environment Variables button.

    On Windows XP, open the Start menu and go to Control Panel | System. Click on the Advanced tab. Click on the Environment Variables button.

    Now, under System variables, select an existing environment variable, such as Path, and click on the Edit button. Alternatively, make a new environment variable by clicking on the New button. Edit the variable's name and value as needed. For example, if we want to add C:\android-sdk\platform-tools and C:\android-sdk\tools to Path, we should append ;C:\android-sdk\platform-tools;C:\android-sdk\tools to the existing value of Path. Note the use of semicolons as separators.

    To apply the changes, click on all the OK buttons until we are back in the main window of Control Panel. Now, log out and again log in.

    Editing environment variables on Mac

    Edit ~/.profile.

    To append to an existing environment variable, add a line such as export PATH=$PATH:~/android-sdk/platform-tools:~/android-sdk/tools. This example appends ~/android-sdk/platform-tools and ~/android-sdk/tools to PATH. Note the use of colons as separators.

    To create a new environment variable, add a line such as export NDKROOT=~/android-ndk.

    Save your changes, log out, and again log in.

    Editing environment variables on Ubuntu

    Edit ~/.pam_environment.

    To append to an existing environment variable, add a line such as PATH DEFAULT=${PATH}:~/android-sdk/platform-tools:~/android-sdk/tools. This example appends ~/android-sdk/platform-tools and ~/android-sdk/tools to PATH. Note the use of colons as separators.

    To create a new environment variable, add a line such as NDKROOT DEFAULT=~/android-ndk.

    Save your changes, log out, and again log in.

Now, we have an Android development environment but we still need OpenCV. We may choose to download a prebuilt version of OpenCV or we may build it from source. These options are discussed in the following two subsections.

Tip

Generally, Android applications should use a prebuilt version of OpenCV. One important reason is that the prebuilt versions are available for Android users as shared libraries, which save disk space and simplify updates.

For the purpose of this book's project, there is no need to build OpenCV from source. We just mention this option for completeness, since it may be of interest to advanced users who want to modify OpenCV.

Getting the prebuilt OpenCV4Android

The prebuilt versions of OpenCV4Android can be downloaded from http://sourceforge.net/projects/opencvlibrary/files/opencv-android/. Look for files that have opencv-android in the name, such as OpenCV-2.4.5-android-sdk.zip (the latest version at the time of writing). Download the latest version and unzip it to any destination, which we will refer to as <opencv>.

Building OpenCV4Android from source

Alternatively, the process for building OpenCV4Android from trunk (the latest, unstable source code) is documented at http://code.opencv.org/projects/opencv/wiki/Building_OpenCV4Android_from_trunk. For a summary of the process for building from trunk, continue reading this section. Otherwise, skip ahead to Building the OpenCV samples with Eclipse, later in this chapter.

Tip

Since trunk contains the latest and unstable source code, there is no guarantee that the build process will succeed. You may need to do your own troubleshooting if you want to build from trunk.

To build OpenCV from source, we need the following additional software:

  • Git: It is a Source Control Management (SCM) tool, which we will use to obtain OpenCV's source code. On Windows or Mac, download and install Git from http://git-scm.com/. On Linux, install it using your package manager. For example, on Ubuntu, open Terminal and run $ sudo apt-get install git-core.

  • CMake: It is a set of build tools. On Windows or Mac, download and install CMake from http://www.cmake.org/cmake/resources/software.html. On Linux, install it using your package manager. For example, on Ubuntu, open Terminal and run $ sudo apt-get install cmake.

  • Apache Ant 1.8.0 or greater: It is a set of build tools for Java. On Linux, just install Ant using your package manager. For example, on Ubuntu, open Terminal and run $ sudo apt-get install ant. On Windows or Mac, download Ant from http://ant.apache.org/bindownload.cgi and unzip it to any destination, which we will refer to as <ant>. Make the following changes to your environment variables:

    • Add <ant>/bin to PATH.

    • Create a variable, ANT_HOME, with the value <ant>.

  • Python 2.6 or greater (but not 3.0 or greater): It is a scripting language that is used by some of the OpenCV build scripts. An appropriate version of Python comes preinstalled on Mac and most Linux systems, including Ubuntu. On Windows, download and install Python from http://www.python.org/getit/. If you have installed multiple versions of Python on your system, ensure that an installation of Python 2.6 or greater (but not 3.0 or greater) is the only one in Path (Windows) or PATH (Mac, Linux, or other Unix-like systems). The OpenCV build scripts do not run properly with Python 3.0 or greater.

Once we have these prerequisites, we may download the OpenCV source code to any location, which we will refer to as <opencv_source>. Then, we may build it using an included script. The steps are platform-specific, and are described as follows:

On Windows, copy <opencv>\android\scripts\wincfg.cmd.tmpl to <opencv>\android\scripts\wincfg.cmd. Edit <opencv>\android\scripts\wincfg.cmd. The locations of several of the prerequisites are declared in this file. Modify them so that they are correct for your system. Save your changes. Then, open Git Bash (Git's command prompt) and run the following commands:

$ git clone git://code.opencv.org/opencv.git <opencv_source>
$ cd <opencv_source>/android
$ scripts/cmake_android.cmd
$ cd build
$ make -j8

On Mac, Ubuntu, or other Unix-like systems, open Terminal (or another command line shell) and run the following commands:

$ git clone git://code.opencv.org/opencv.git <opencv_source>
$ cd <opencv_source>/android
$ sh ./scripts/cmake_android.sh
$ cd build
$ make -j8

If all goes well, we should get a build of OpenCV4Android in <opencv_source>/android/build. We may move it elsewhere if we wish. We will refer to its final location as <opencv>.

 

Building the OpenCV samples with Eclipse


Building and running a few sample applications is a good way to test that OpenCV is correctly set up. At the same time, we can practice using Eclipse.

Let's start by launching Eclipse. The Eclipse launcher should be located at <eclipse>/eclipse.exe (Windows), <eclipse>/Eclipse.app (Mac), or <eclipse>/eclipse (Linux). Run it.

We should see a window called Workspace Launcher, which asks us to select a workspace. A workspac e is the root directory for a set of related Eclipse projects. If we are using TADP, enter <tadp>/nvsample_workspace, which is a workspace where the OpenCV4Android library, samples, and tutorials are already set up as projects. Otherwise, enter any location you choose.

Tip

We can return to Workspace Launcher anytime via the menu: File | Switch Workspace | Other….

If the Welcome to Eclipse screen appears, click on the Workbench button.

Now, we should see a window with several panels, including Package Explorer. If we are not using TAPD, we need to import the OpenCV sample projects into our new workspace. Right-click on Package Explorer and select Import… from the context menu.

The Import window should appear. Select General | Existing Projects into Workspace, and then click on Next>.

On the second page of the Import window, enter <opencv> in the Select root directory: field. Under the Projects: label, a list of detected projects should appear (If not, click on Refresh). The list should include OpenCV library, samples, and tutorials. Ensure that all projects are selected and click on Finish to import them.

Once the projects are imported, we may need to fix some configuration issues. Our development environment may have different paths, and different versions of the Android SDK, than the ones in the samples' default configuration.

Any resulting errors will be reported in the Problems tab.

Tip

We should start by resolving any errors in the OpenCV Library project, as the samples and tutorials depend on the library.

The following are some of the common configuration problems, and their symptoms and solutions:

  • The target Android version might not be properly specified. The symptoms are that imports from the java and android packages fail, and there are error messages such as The project was not built since its build path is incomplete. The solution is to right-click on the project in Package Explorer, select Properties from the context menu, select the Android section, and checkmark one of the available Android versions. These steps should be repeated for all projects. At compile time, OpenCV and its samples must target Android 3.0 (API level 11) or greater, though at runtime they also support Android 2.2 (API level 8) or greater.

  • If we are working on Mac or Linux, the C++ samples might be misconfigured to use the Windows build executable. The symptom is an error message such as Program "{ndk}/ndk-build.cmd" not found in PATH. The solution is to right-click on the project in Package Explorer, select Properties from the context menu, select the C/C++ Build section, and edit the Build command: field to remove the .cmd extension. These steps should be repeated for all the native (C++) projects, which include OpenCV Sample - face-detection and OpenCV Tutorial 2 - Mixed Processing.

    Tip

    If we are using the projects in TADP's <tadp>/nvsample_workspace, we only need to troubleshoot the projects that have names starting with OpenCV. For this book's purposes, the other TADP samples are not relevant.

Once the OpenCV projects no longer show any errors, we can prepare to test them on an Android device. Recall that the device must have Android 2.2 (Froyo) or a greater version, and a camera. To enable Eclipse to communicate with the device, we must enable the device's USB debugging option with the help of the following steps:

  1. Open the Settings app.

  2. On Android 4.2 or greater, go to the About phone or About tablet section and tap Build number seven times. This step enables the Developer options section.

  3. Go to the Developer options section (on Android 4.0 or greater) or the Applications | Development section (on Android 3.2 or less). Enable the USB debugging option.

Now, open the Play Store app, and find and install the OpenCV Manager app. (The app's page in the Play Store should look similar to the previous screenshot.) OpenCV Manager takes care of checking for any OpenCV library updates when we run any OpenCV applications.

Tip

If you do not have the Play Store app on your device, then you need to install OpenCV Manager and certain OpenCV libraries via USB as per the instructions at http://docs.opencv.org/android/service/doc/UseCases.html.

Now, we must prepare our main computer for communication with the Android device. The required steps vary, depending on our operating system.

On Windows, we need to install the proper USB drivers for the Android device. Different vendors and devices have different drivers. The official Android documentation provides links to the various vendors' driver download sites at http://developer.android.com/tools/extras/oem-usb.html#Drivers.

On Linux, before connecting an Android device via USB, we must specify the device's vendor in a permissions file. Each vendor has a unique ID number, as listed in the official Android documentation at http://developer.android.com/tools/device.html#VendorIds. We will refer to this ID number as <vendor_id>. To create the permissions file, open a command prompt application (such as Terminal) and run the following commands:

$ cd /etc/udev/rules.d/
$ sudo touch 51-android.rules
$ sudo chmod a+r 51-android-rules

Note that the permissions file needs to have root ownership, so we use sudo when creating or modifying it. Now, open the file in an editor such as gedit:

$ sudo gedit 51-android-rules

For each vendor, append a new line to the file. Each of these lines should have the following format:

SUBSYSTEM=="usb", ATTR{idVendor}=="<vendor_id>", MODE="0666", GROUP="plugdev"

Save the permissions file and quit the editor.

On Mac, no special drivers or permissions are required.

Plug the Android device into your computer's USB port. In Eclipse, select one of the OpenCV sample projects in Package Explorer. Then, from the menu system, select Run | Run as… | Android Application.

An Android Device Chooser window should appear. Your Android device should be listed under Choose a running Android device. (If the device is not listed, try unplugging it and plugging it back in. If that does not work, also try disabling and re-enabling the device's USB debugging option, as described earlier.)

Select the device and click on OK.

If the Auto Monitor Logcat window appears, select the Yes radio button and the verbose drop-down option, and click on OK. This option ensures that all the log output from the application will be visible in Eclipse.

On the Android device you might get a message: OpenCV library package was not found! Try to install it? Make sure the device is connected to the Internet and then touch the Yes button on your device. The Play Store will open to show an OpenCV package. Install the package and then press the hardware back button to return to the sample application, which should be ready for use.

For OpenCV 2.4.3.2, the samples and tutorials have the following functionality:

  • Sample – 15 puzzle: It splits up a camera feed to make a sliding-block puzzle. The user may swipe blocks to move them.

  • Sample – color-blob-detection: It detects color regions in a camera feed. The user may touch anywhere to see the outline of a color region.

  • Sample – face-detection: It draws green rectangles around faces in a camera feed.

  • Sample – image-manipulations: It applies filters to a camera feed. The user may press the Android menu button to select from a list of filters.

  • Tutorial 1 – Add OpenCV: It displays a camera feed. The user may press the … menu to select a different camera feed implementation (Java or native C++).

  • Tutorial 2 – Use OpenCV Camera: It applies filters to a camera feed. The user may press the … menu to select from a list of filters.

  • Tutorial 3 – Add Native OpenCV: It draws red circles around interest points or features in a camera feed. Generally speaking, interest points or features lie along the high-contrast edges in an image. They are potentially useful in image recognition and tracking applications.

  • Tutorial 4 – Mix Java+Native OpenCV: It combines the functionality of Tutorial 2 and Tutorial 3. The user may press the … menu to select from a list of filters and a preview of interest points.

  • Tutorial 5 – Camera Control: It applies filters to a camera feed, which has a customizable resolution. The user may press the … menu to select from a list of filters and a list of resolutions.

Try these applications on your Android device! While an application is running, its log output should appear in the LogCat tab in Eclipse.

Feel free to browse the projects' source code via Package Explorer, to see how they were made. Alternatively, you might want to return to the official samples and tutorials later, after we have built our own project over the course of this book.

 

Finding documentation and help


The OpenCV Java API and C++ API are both relevant to Android. The Java API documentation is online at http://docs.opencv.org/java/. The C++ API documentation is online at http://docs.opencv.org/. The following documents, which mostly use C++ code, are also available as downloadable PDF files:

If the documentation does not seem to answer your question, try talking to the OpenCV community. Here are some sites where you will find helpful people:

Also, you can read or submit bug reports at http://code.opencv.org/projects/opencv/issues?query_id=4. Finally, if you need to take your issue to the highest authority, you can email the OpenCV4Android developers at .

 

Summary


By now, we should have an Android and OpenCV development environment that can do everything we need for the project described in this book. Depending on which approach we took, we might also have a set of tools that we can use to reconfigure and rebuild OpenCV for our future needs.

We know how to build the OpenCV Android samples in Eclipse. These samples cover a different range of functionality than this book's project, but they are useful as additional learning aids. We also know where to find documentation and help.

About the Author

  • Joseph Howse

    Joseph Howse lives in a Canadian fishing village with four cats; the cats like fish, but they prefer chicken.

    Joseph provides computer vision expertise through his company, Nummist Media. His books include OpenCV 4 for Secret Agents, OpenCV 3 Blueprints, Android Application Programming with OpenCV 3, iOS Application Development with OpenCV 3, Learning OpenCV 3 Computer Vision with Python, and Python Game Programming by Example, published by Packt.

    Browse publications by this author

Latest Reviews

(3 reviews total)
Excellent
Good
Excellent