TensorFlow for Mobile and IoT and TensorFlow.js

In this chapter we will learn the basics of TensorFlow for Mobile and IoT (Internet of Things). We will briefly present TensorFlow Mobile and we will introduce TensorFlow Lite in more detail. TensorFlow Mobile and TensorFlow Lite are open source deep learning frameworks for on-device inference. Some examples of Android, iOS, and Raspberry PI applications will be discussed, together with examples of deploying pretrained models such as MobileNet v1, v2, v3 (image classification models designed for mobile and embedded vision applications), PoseNet for pose estimation (a vision model that estimates the poses of people in image or video), DeepLab segmentation (an image segmentation model that assigns semantic labels (for example, dog, cat, car) to every pixel in the input image), and MobileNet SSD object detection (an image classification model that detects multiple objects with bounding boxes). This chapter will conclude with an example...

TensorFlow Mobile

TensorFlow Mobile is a framework for producing code on iOS and Android. The key idea is to have a platform that allows you to have light models that don't consume too much device resources such as battery or memory. Typical examples of applications are image recognition on the device, speech recognition, or gesture recognition. TensorFlow Mobile was quite popular until 2018 but then became progressively less and less adopted in favor of TensorFlow Lite.

TensorFlow Lite

TensorFlow Lite is a lightweight platform designed by TensorFlow. This platform is focused on mobile and embedded devices such as Android, iOS, and Raspberry PI. The main goal is to enable machine learning inference directly on the device by putting a lot of effort in three main characteristics: (1) small binary and model size to save on memory, (2) low energy consumption to save on the battery, and (3) low latency for efficiency. It goes without saying that battery and memory are two important resources for mobile and embedded devices. In order to achieve these goals, Lite uses a number of techniques such as Quantization, FlatBuffers, Mobile interpreter, and Mobile converter, which we are going to review briefly in the following sections.

Quantization

Quantization refers to a set of techniques that constrains an input made of continuous values (such as real numbers) into a discrete set (such as integers). The key idea is to reduce the space occupancy of Deep...

Pretrained models in TensorFlow Lite

In many interesting use cases, it is possible to use a pretrained model that is already suitable for mobile computation. This is a field of active research with new proposals coming pretty much every month. TensorFlow Lite comes with a set of prebuilt models that are ready to use (https://www.tensorflow.org/lite/models/). As of October 2019, these include:

• Image classification: Used to identify multiple classes of objects such as places, plants, animals, activities, and people.
• Object detection: Used to detect multiple objects with bounding boxes.
• Pose estimation: Used to estimate poses with single or multiple people.
• Smart reply: Used to create reply suggestions for conversational chat messages.
• Segmentations: Identifies the shape of objects together with semantic labels for people, places, animals, and many additional classes.
• Style transfers: Used to apply artistic styles to any given image.
• Text classification...

An overview of federated learning at the edge

As discussed, edge computing is a distributed computing model that brings computation and data closer to the location where it is needed.

Now, let's introduce Federated Learning (FL) [8] at the edge, starting with two use cases.

Suppose you built an app for playing music on mobile devices and then you want to add recommendation features aimed at helping users to discover new songs they might like. Is there a way to build a distributed model that leverages each user's experience without disclosing any private data?

Suppose you are a car manufacturer producing millions of cars connected via 5G networks, and then you want to build a distributed model for optimizing each car's fuel consumption. Is there a way to build such a model without disclosing the driving behavior of each user?

Traditional machine learning requires you to have a centralized repository for training data either on your desktop, or in your datacenter...

TensorFlow.js

TensorFlow.js is a JavaScript library for machine learning models that can work either in vanilla mode or via Node.js. In this section we are going to review both of them.

Vanilla TensorFlow.js

TensorFlow.js is a JavaScript library for training and using Machine Learning (ML) models in a browser. It is derived from deeplearn.js, an open source, hardware-accelerated library for doing Deep Learning (DL) in JavaScript, and is now a companion library to TensorFlow.

The most common use of TensorFlow.js is to make pretrained ML/DL models available on the browser. This can help in situations where it may not be feasible to send client data back to the server due to network bandwidth or security concerns. However, TensorFlow.js is a full stack ML platform, and it is possible to build and train an ML/DL model from scratch, as well as fine-tune an existing pretrained model with new client data.

An example of a TensorFlow.js application is the TensorFlow Projector (https...

Summary

In this chapter we have discussed how to use TensorFlow Lite for mobile devices and IoT and deployed real applications on Android devices. Then, we also talked about Federated Learning for distributed learning across thousands (millions) of mobile devices, taking into account privacy concerns. The last section of the chapter was devoted to TensorFlow.js for using TensorFlow with vanilla JavaScript or with Node.js.

The next chapter is about AutoML, a set of techniques used to enable domain experts who are unfamiliar with machine learning technologies to use ML techniques easily.

References

1. Quantization-aware training https://github.com/tensorflow/tensorflow/tree/r1.13/tensorflow/contrib/quantize
2. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko (Submitted on 15 Dec 2017); https://arxiv.org/abs/1712.05877
3. MobileNetV2: Inverted Residuals and Linear Bottlenecks, Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen (Submitted on 13 Jan 2018 (v1), last revised 21 Mar 2019 (v4)) https://arxiv.org/abs/1806.08342
4. MnasNet: Platform-Aware Neural Architecture Search for Mobile, Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. Le https://arxiv.org/abs/1807.11626
5. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos...