Intelligent Mobile Projects with TensorFlow

4 (1 reviews total)
By Jeff Tang
    What do you get with a Packt Subscription?

  • Instant access to this title and 7,500+ eBooks & Videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Free Chapter
    Getting Started with Mobile TensorFlow
About this book
As a developer, you always need to keep an eye out and be ready for what will be trending soon, while also focusing on what's trending currently. So, what's better than learning about the integration of the best of both worlds, the present and the future? Artificial Intelligence (AI) is widely regarded as the next big thing after mobile, and Google's TensorFlow is the leading open source machine learning framework, the hottest branch of AI. This book covers more than 10 complete iOS, Android, and Raspberry Pi apps powered by TensorFlow and built from scratch, running all kinds of cool TensorFlow models offline on-device: from computer vision, speech and language processing to generative adversarial networks and AlphaZero-like deep reinforcement learning. You’ll learn how to use or retrain existing TensorFlow models, build your own models, and develop intelligent mobile apps running those TensorFlow models. You'll learn how to quickly build such apps with step-by-step tutorials and how to avoid many pitfalls in the process with lots of hard-earned troubleshooting tips.
Publication date:
May 2018


Getting Started with Mobile TensorFlow

This chapter covers how to get your development environments set up for building all the iOS or Android apps with TensorFlow that are discussed in the rest of the book. We won't discuss in detail all the supported TensorFlow versions, OS versions, Xcode, and Android Studio versions that can be used for development, as that kind of information can easily be found on the TensorFlow website ( or via Google. Instead, we'll just talk briefly about sample working environments in this chapter so that we can dive in quickly to look at all the amazing apps you can build with the environments.

If you already have TensorFlow, Xcode, and Android Studio installed, and can run and test the sample TensorFlow iOS and Android apps, and if you already have an NVIDIA GPU installed for faster deep learning model training, you can skip this chapter. Or you can jump directly to the section that you're unfamiliar with.

We're going to cover the following topics in this chapter (how to set up the Raspberry Pi development environment will be discussed in Chapter 12, Developing TensorFlow Apps on Raspberry Pi):

  • Setting up TensorFlow
  • Setting up Xcode
  • Setting up Android Studio
  • TensorFlow Mobile vs TensorFlow Lite
  • Running sample TensorFlow iOS apps
  • Running sample TensorFlow Android apps

Setting up TensorFlow

TensorFlow is the leading open source framework for machine intelligence. When Google released TensorFlow as an open source project in November 2015, there were already several other similar open source frameworks for deep learning: Caffe, Torch, and Theano. By Google I/O 2018 on May 10, TensorFlow on GitHub has reached 99k stars, an increase of 14k stars in 4 months, while Caffe has increased only 2k to 24k stars. Two years later, it's already the most popular open source framework for training and deploying deep learning models (it also has good support for traditional machine learning). As of January 2018, TensorFlow has close to 85k stars ( on GitHub, while the other three leading open source deep learning frameworks, Caffe (, CNTK (, and Mxnet ( have over 22k, 13k, and 12k stars, respectively.

If you're a little confused about the buzz words machine learning, deep learning, machine intelligence, and artificial intelligence (AI), here's a quick summary: machine intelligence and AI are really just the same thing; machine learning is a field, also the most popular one, of AI; deep learning is one special type of machine learning, and is also the modern and most effective approach to solving complicated problems such as computer vision, speech recognition and synthesis, and natural language processing. So in this book, when we say AI, we primarily mean deep learning, the savior that took AI from the long winter to the summer. For more information about the AI winter and deep learning, you can check out and

We assume you already have a basic understanding of TensorFlow, but if you don't, you should check out the Get Started ( and Tutorials ( parts of the TensorFlow website or the Awesome TensorFlow tutorials ( Two good books on the topic are Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow, 2nd Edition by Sebastian Raschka and Vahid Mirjalili, and Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron.

TensorFlow can be installed on MacOS, Ubuntu or Windows. We'll cover the steps to install TensorFlow 1.4 from its source on MacOS X El Capitan (10.11.6), macOS Sierra (10.12.6), and Ubuntu 16.04. If you have a different OS or version, you can refer to the TensorFlow Install ( documentation for more information. By the time you read this book, it's likely a newer TensorFlow version will come out. Although you should still be able to run the code in the book with the newer version, it's not a guarantee, which is why we use the TensorFlow 1.4 release source code to set up TensorFlow on Mac and Ubuntu; that way, you can easily test run and play with the apps in the book.

Since we wrote the paragraph above in December 2017, there have been four new official releases of TensorFlow (1.5, 1.6, 1.7, and 1.8), which you can download at or from the TensorFlow source code repo (, and a new version of Xcode (9.3) as of May 2018. Newer versions of TensorFlow, such as 1.8, by default support newer versions of NVIDIA CUDA and cuDNN (see the section Setting up TensorFlow on GPU-powered Ubuntu for detail), and you'd better follow the official TensorFlow documentation to install the latest TensorFlow version with GPU support. In this and the following chapters, we may refer to a specific TensorFlow version as an example, but will keep all iOS, Android, and Python code tested and, if needed, updated for the latest TensorFlow, Xcode, and Android Studio versions in the book's source code repo at

Overall, we'll use TensorFlow on Mac to develop iOS and Android TensorFlow apps, and TensorFlow on Ubuntu to train deep learning models used in the apps.

Setting up TensorFlow on MacOS

Generally, you should use a virtualenv, Docker, or Anaconda installation to install TensorFlow in an isolated environment. But as we have to build iOS and Android TensorFlow apps using the TensorFlow source code, we might as well build TensorFlow itself from source, in which case, using the native pip installation choice could be easier than other options. If you want to experiment with different TensorFlow versions, we recommend you install other TensorFlow versions using one of the virtualenv, Docker, and Anaconda options. Here, we'll have TensorFlow 1.4 installed directly on your MacOS system using the native pip and Python 2.7.10.

Follow these steps to download and install TensorFlow 1.4 on MacOS:

  1. Download the TensorFlow 1.4.0 source code (zip or tar.gz) from the TensorFlow releases page on GitHub:
  2. Uncompress the downloaded file and drag the tensorflow-1.4.0 folder to your home directory
  3. Make sure you have Xcode 8.2.1 or above installed (if not, read the Setting up Xcode section first)
  4. Open a new Terminal window, then cd tensorflow-1.4.0
  5. Run xcode-select --install to install command-line tools
  6. Run the following commands to install other tools and packages needed to build TensorFlow:
sudo pip install six numpy wheel  
brew install automake 
brew install libtool 
brew upgrade bazel 
  1. Build from the TensorFlow source with CPU-only support (we'll cover the GPU support in the next section) and generate a pip package file with the .whl file extension:
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package 
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg 
  1. Install the TensorFlow 1.4.0 CPU package:
sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-1.4.0-cp27-cp27m-macosx_10_12_intel.whl

If you encounter any error during the process, googling the error message, to be honest, should be the best way to fix it, as we intend in this book to focus on the tips and knowledge, not easily available elsewhere, gained from our long hours of building and debugging practical mobile TensorFlow apps. One particular error you may see is the Operation not permitted error when running the sudo pip install commands. To fix it, you can disable your Mac's System Integrity Protection (SIP) by restarting the Mac and hitting the Cmd + R keys to enter the recovery mode, then under Utilities-Terminal, running csrutil disable before restarting Mac. If you're uncomfortable with disabling SIP, you can follow the TensorFlow documentation to try one of the more complicated installation methods such as virtualenv.

If everything goes well, you should be able to run on your Terminal Window, Python or preferably IPython, then run import tensorflow as tf and tf.__version__ to see 1.4.0 as output.

Setting up TensorFlow on GPU-powered Ubuntu

One of the benefits of using a good deep learning framework, such as TensorFlow, is its seamless support for using a Graphical Processing Unit (GPU) in model training. It'd be a lot faster training a non-trivial TensorFlow-based model on a GPU than on a CPU, and currently NVIDIA offers the best and most cost-effective GPUs supported by TensorFlow. And Ubuntu is the best OS for running NVIDIA GPUs with TensorFlow. You can easily buy one GPU for a few hundred bucks and install it on an inexpensive desktop with an Ubuntu system. You can also install NVIDIA GPU on Windows but TensorFlow's support for Windows is not as good as that for Ubuntu.

To train the models deployed in the apps in this book, we use NVIDIA GTX 1070, which you can purchase on Amazon or eBay for about $400. There's a good blog by Tim Dettmers on which GPUs to use for deep learning ( After you get such a GPU and install it on your Ubuntu system, and before you install the GPU-enabled TensorFlow, you need to install NVIDIA CUDA 8.0 (or 9.0) and the cuDNN (CUDA-Deep Neural Network library) 6.0 (or 7.0), both are supported by TensorFlow 1.4.

An alternative to setting up your own GPU-powered Ubuntu with TensorFlow is to use TensorFlow in a GPU-enabled cloud service such as Google Cloud Platform's Cloud ML Engine ( There are pros and cons of each option. Cloud services are generally time-based billing. If your goal is to train or retrain models to be deployed on mobile devices, meaning the models are not super complicated, and if you plan to do machine learning training for a long time, it'd be more cost effective and satisfying to have your own GPU.

Follow these steps to install CUDA 8.0 and cuDNN 6.0 on Ubuntu 16.04 (you should be able to download and install CUDA 9.0 and cuDNN 7.0 in a similar way):

  1. Find the NVIDIA CUDA 8.0 GA2 release at, and make the selections shown in the following screenshot:
Fig 1.1 Getting ready to download CUDA 8.0 on Ubuntu 16.04
  1. Download the base installer as shown in the following screenshot:
Fig 1.2 Choosing the CUDA 8.0 installer file for Ubuntu 16.04
  1. Open a new Terminal and run the following commands (you'll also need to add the last two commands to your .bashrc file for the two environment variables to take effect next time you launch a new Terminal):
sudo dpkg -i /home/jeff/Downloads/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb  
sudo apt-get update 
sudo apt-get install cuda-8-0 
export CUDA_HOME=/usr/local/cuda 
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
  1. Download the NVIDIA cuDNN 6.0 for CUDA 8.0 at – you'll be asked to sign up (for free) with an NVIDIA Developer account first before you can download it, as shown in the next screenshot (choose the highlighted cuDNN v6.0 Library for Linux):
Fig 1.3 Selecting cuDNN 6.0 for CUDA 8.0 on Linux
  1. Unzip the downloaded file, assuming it's under the default ~/Downloads directory, and you'll see a folder named cuda, with two subfolders named include and lib64
  2. Copy the cuDNN include and lib64 files to the CUDA_HOME's lib64 and include folders:
sudo cp ~/Downloads/cuda/lib64/* /usr/local/cuda/lib64 
sudo cp ~/Downloads/cuda/include/cudnn.h /usr/local/cuda/include 

Now we're ready to install the GPU-enabled TensorFlow 1.4 on Ubuntu (the first two steps given here are the same as those described in the section Setting up TensorFlow on MacOS):

  1. Download the TensorFlow 1.4.0 source code (zip or tar.gz) from the TensorFlow releases page on GitHub:
  2. Uncompress the downloaded file and drag the folder to your home directory
  3. Download the bazel installer, at
  4. Open a new Terminal Window, then run the following commands to install the tools and packages needed to build TensorFlow:
sudo pip install six numpy wheel  
cd ~/Downloads 
chmod +x 
./ --user
  1. Build from the TensorFlow source with GPU-enabled support and generate a pip package file with the .whl file extension:
cd ~/tensorflow-1.4.0 
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package 
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg 
  1. Install the TensorFlow 1.4.0 GPU package:
sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-1.4.0-cp27-cp27mu-linux_x86_64.whl 

Now, if all goes well, you can launch IPython and enter the following scripts to see the GPU information TensorFlow is using:

In [1]: import tensorflow as tf 
In [2]: tf.__version__ 
Out[2]: '1.4.0' 
In [3]: sess=tf.Session() 
2017-12-28 23:45:37.599904: I tensorflow/stream_executor/cuda/] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 
2017-12-28 23:45:37.600173: I tensorflow/core/common_runtime/gpu/] Found device 0 with properties:  
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7845 
pciBusID: 0000:01:00.0 
totalMemory: 7.92GiB freeMemory: 7.60GiB 
2017-12-28 23:45:37.600186: I tensorflow/core/common_runtime/gpu/] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1) 

Congratulations! You're now ready to train the deep learning models used in the apps in this book. Before we start having fun with our new toy and use it to train our cool models and then deploy and run them on mobile devices, let's first see what it takes to be ready for developing mobile apps.


Setting up Xcode

Xcode is used to develop iOS apps and you need a Mac computer and a free Apple ID to download and install it. If your Mac is relatively older and with OS X El Capitan (version 10.11.6), you can download Xcode 8.2.1 at Or if you have macOS Sierra (version 10.12.6) or later, installed, you can download Xcode 9.2 or 9.3, the latest version as of May 2018, from the preceding link. All the iOS apps in the book have been tested in both Xcode 8.2.1, 9.2, and 9.3.

To install Xcode, simply double-click the downloaded file and follow the steps on the screen. It's pretty straightforward. You can now run apps on the iOS simulator that comes with Xcode or your own iOS device. Starting Xcode 7, you can run and debug your iOS apps on an iOS device for free, but if you want to distribute or publish your apps, you need to enroll in the Apple Developer Program for $99 a year as an individual:

Although you can test run many apps in the book with the Xcode simulator, some apps in the book require the camera on your actual iOS device to take a picture before processing it with a deep learning model trained with TensorFlow. In addition, it's generally better to test a model on an actual device for accurate performance and memory usage: a model that runs fine in the simulator may crash or run too slow in an actual device. So it's strongly recommended or required that you test and run the iOS apps in the book on your actual iOS device at least once, if not always.

This book assumes you're comfortable with iOS programming, but if you're new to iOS development, you can learn from many excellent online tutorials such as the iOS tutorials by Ray Wenderlich ( We won't cover complicated iOS programming; we'll mainly show you how to use the TensorFlow C++ API in our iOS apps to run the TensorFlow trained models to perform all kinds of intelligent tasks. Both Objective-C and Swift code, the two official iOS programming languages from Apple, will be used to interact with the C++ code in our mobile AI apps.


Setting up Android Studio

Android Studio is the best tool for Android app development, and TensorFlow has great support for using it. Unlike Xcode, you can install and run Android Studio on Mac, Windows, or Linux. For detailed system requirements, see the Android Studio website ( Here, we'll just cover how to set up Android Studio 3.0 or 3.0.1 on Mac – all the apps in the book have been tested on both versions.

First, download the Android Studio 3.0.1, or the latest version if it's newer than 3.0.1 and if you don't mind fixing possible minor issues, from the preceding link. You can also download 3.0.1 or 3.0 from its archives at

Then, double-click the downloaded file and drag and drop the Android icon to Applications. If you have a previously installed Android Studio, you'll get a prompt asking you if you want to replace it with the newer one. You can just select Replace.

Next, open Android Studio and you need to provide the path to the Android SDK, which by default is in ~/Library/Android/sdk if you have a previous version of Android Studio installed, or you can select Open an existing Android Studio project, then go to the TensorFlow 1.4 source directory created in the section Setting up TensorFlow on MacOS, and open the tensorflow/examples/android folder. After that, you can download the Android SDK by either clicking the link to an Install Build Tools message or going to Android Studio's Tools | Android | SDK Manager, as shown in the following screenshot. From the SDK Tools tab there, you can check the box next to a specific version of Android SDK Tools and click the OK button to install that version:

Fig 1.4 Android SDK Manager to install SDK tools and NDK

Finally, as TensorFlow Android apps use the native TensorFlow library in C++ to load and run TensorFlow models, you need to install the Android Native Development Kit (NDK), which you can do either from the Android SDK Manager shown in the preceding screenshot, or by downloading NDK directly from Both the NDK version r16b and r15c have been tested to run the Android apps in the book. If you download the NDK directly, you may also need to set the Android NDK location after opening your project and selecting Android Studio's File | Project Structure, as shown in the following screenshot:

Fig 1.5 Setting project-level Android NDK location

With both Android SDK and NDK installed and set up, you're ready to test run sample TensorFlow Android apps.


TensorFlow Mobile vs TensorFlow Lite

Before we start running sample TensorFlow iOS and Android apps, let's clarify one big picture. TensorFlow currently has two approaches to developing and deploying deep learning apps on mobile devices: TensorFlow Mobile and TensorFlow Lite. TensorFlow Mobile was part of TensorFlow from the beginning, and TensorFlow Lite is a newer way to develop and deploy TensorFlow apps, as it offers better performance and smaller app size. But there's one key factor that will let us focus on TensorFlow Mobile in this book, while still covering TensorFlow Lite in one chapter: TensorFlow Lite is still in developer preview as of TensorFlow 1.8 and Google I/O 2018 in May 2018. So to develop production-ready mobile TensorFlow apps now, you have to use TensorFlow Mobile, as recommended by Google.

Another reason we decided to focus on TensorFlow Mobile now is while TensorFlow Lite only offers a limited support for model operators, TensorFlow Mobile supports customization to add new operators not supported by TensorFlow Mobile by default, which you'll see happens pretty often in our various models of different AI apps.

But in the future, when TensorFlow Lite is out of developer preview, it's likely to replace TensorFlow Mobile, or at least overcome its current limitations. To get yourself ready for that, we'll cover TensorFlow Lite in detail in a later chapter.


Running sample TensorFlow iOS apps

In the last two sections of this chapter, we'll test run three sample iOS apps and four sample Android apps that come with TensorFlow 1.4 to make sure you have your mobile TensorFlow development environments set up correctly and give you a quick preview at what some TensorFlow mobile apps can do.

The source code of the three sample TensorFlow iOS apps is located at tensorflow/examples/ios: simple, camera, and benchmark. To successfully run these samples, you need to first download one pretrained deep learning model by Google, called Inception (, for image recognition. There are several versions of Inception: v1 to v4, with better accuracy in each newer version. Here we'll use Inception v1 as the samples were developed for it. After downloading the model file, copy the model-related files to each of the samples' data folder:

curl -o ~/graphs/ 
unzip ~/graphs/ -d ~/graphs/inception5h 
cd tensorflow/examples/ios 
cp ~/graphs/inception5h/* simple/data/ 
cp ~/graphs/inception5h/* camera/data/ 
cp ~/graphs/inception5h/* benchmark/data/ 

Now, go to each app folder and run the following commands to download the required pod for each app before opening and running the apps:

cd simple 
pod install 
open tf_simple_example.xcworkspace 
cd ../camera 
pod install 
open tf_camera_example.xcworkspace 
cd ../benchmark 
pod install 
open tf_benchmark_example.xcworkspace  

You can then run the three apps on an iOS device, or the simple and benchmark apps on an iOS simulator. If you tap the Run Model button after running the simple app, you'll see a text message saying that the TensorFlow Inception model is loaded, followed by several top recognition results along with confidence values.

If you tap the Benchmark Model button after running the benchmark app, you'll see the average time it takes to run the model for over 20 times. For example, it takes an average of about 0.2089 seconds on my iPhone 6, and 0.0359 seconds on the iPhone 6 simulator.

Finally, running the camera app on an iOS device and pointing the device camera around shows you the objects the app sees and recognizes in real time.


Running sample TensorFlow Android apps

There are four sample TensorFlow Android apps named TF Classify, TF Detect, TF Speech, and TF Stylize, located in tensorflow/examples/android. The easiest way to run these samples is to just open the project in the preceding folder using Android Studio, as shown in the Setting up Android Studio section, then make a single change by editing the project's build.gradle file and changing def nativeBuildSystem = 'bazel' to def nativeBuildSystem = 'none'.

Now connect an Android device to your computer and build, install and run the app by selecting Android Studio's Run | Run 'android'. This will install four Android apps with the names TF Classify, TF Detect, TF Speech, and TF Stylize on your device. TF Classify is just like the iOS camera app, using the TensorFlow Inception v1 model to do real-time object classification with the device camera. TF Detect uses a different model, called Single Shot Multibox Detector (SSD) with MobileNet, a new set of deep learning models Google released that are targeted in particular to mobile and embedded devices, to perform object detection, drawing rectangles on detected objects. TF Speech uses another different deep learning (speech recognition) model to listen and recognize a small set of words such as Yes, No, Left, Right, Stop and Go. TF Stylize uses yet another model to change the style of the images the camera sees. For more detailed information on these apps, you can check out the TensorFlow Android example documentation at



In this chapter, we covered how to install TensorFlow 1.4 on Mac and Ubuntu, how to set up a cost effective NVIDIA GPU on Ubuntu for faster model training, and how to set up Xcode and Android Studio for mobile AI app development. We also showed you how to run some cool sample TensorFlow iOS and Android apps. We'll discuss in detail in the rest of the book how to build and train, or retrain each of those models used in the apps, and many others, on our GPU-powered Ubuntu system, and show you how to deploy the models in iOS and Android apps and write the code to use the models in your mobile AI apps. Now that we're all set and ready, we can't wait to hit the road. It'll be an exciting journey, a journey we'd certainly be happy to share with our friends. So why not start with our best friends, and let's see what it takes to build a dog breed recognition app?

About the Author
  • Jeff Tang

    Jeff Tang fell in love with classical AI more than two decades ago. After his MS in CS, he worked on Machine Translation for 2 years and then, to survive the long AI winter, he worked on enterprise apps, voice apps, web apps, and mobile apps at startups, AOL, Baidu, and Qualcomm. He developed a top-selling iOS app with millions of downloads and was recognized by Google as a Top Android Market Developer. He reconnected with modern AI in 2015 and knew that AI will be his passion and commitment for the next two decades. One of his favorite topics is to make AI available anytime anywhere and hence the book.

    Browse publications by this author
Latest Reviews (1 reviews total)
With special discount and feel happily also!
Recommended For You
Intelligent Mobile Projects with TensorFlow
Unlock this book and the full library FREE for 7 days
Start now