Firebase Machine Learning

Machine Learning (ML) has become a critical topic in software development. In a nutshell, ML means that you import data that “trains” a software application and use this data to refine a model. This trained model can then be used to solve problems that would be virtually impossible with traditional programming. To make this process more manageable, Firebase offers a service called Firebase Machine Learning, an ML kit that we could define as “pre-built ML”.

Among other functionalities, it contains text recognition, image labeling, face detection, and barcode scanning. For functionalities that are not already provided by Firebase ML, you can create your own custom model with TensorFlow. Most of the services outlined in this chapter can run both on the cloud and on your device.

In this chapter, you will start by taking a picture with your device; then, you will use Firebase ML to recognize text, barcodes, images, and faces...

Using the device’s camera

In this recipe, you will use the Camera plugin to create a canvas for ML Kit’s vision models. The camera plugin is not related to ML, but being able to take pictures is one of the prerequisites for ML visual functions that you will use in the following recipes in this chapter. Of course, you can also skip this recipe and use images you download from the web if you prefer.

By the end of this recipe, you will be able to use the device’s cameras (front and rear) to take pictures and use them in your apps.

Getting ready

For this recipe, you should create a new project and set up Firebase as described in Chapter 13, Using Firebase, in the Configuring a Firebase app recipe.

How to do it...

In this recipe, you will add the camera functionality to your app. Users will be able to take a picture with the front or rear camera of their device. Follow these steps:

1. In the dependencies section of the project’s...

Recognizing text from an image

We’ll start with ML by incorporating ML Kit’s text recognizer. You will create a feature where you take a picture, and if there is some recognizable text in it, ML Kit will turn it into one or more strings.

Getting ready

For this recipe, you should have completed the previous one: Using the device camera.

How to do it...

In this recipe, after taking a picture, you will add a text recognition feature. Follow these steps:

1. Import the latest version of the google_mlkit_commons and google_mlkit_text_recognition packages in your pubspec.yaml file by typing in your Terminal:

   flutter pub add google_mlkit_commons 
   flutter pub add google_mlkit_text_recognition 
   

2. Create a new file in the lib folder of your project and call it ml.dart.
3. Inside the new file, import the dart:io and google_mlkit_text_recognition packages:

   import 'dart:io'; 
   import ' package:google_mlkit_text_recognition...

Reading a barcode

Adding barcode reading capabilities to your app can open several scenarios for your projects. It is a fast and convenient way to take some user input and return relevant information.

The ML Kit Barcode Scanner library takes an input image and extracts any barcodes that it finds within the image itself.

ML Kit can read most standard barcode formats, including linear and 2D formats:

• Linear: Code 39, Code 93, Code 128, Codabar, EAN-8, EAN-13, ITF, UPC-A, and UPC-E
• 2D: Aztec, Data Matrix, PDF417, and QR code

In this recipe, you will see an easy way to add this feature to the sample app.

Getting ready

Before following this recipe, you should have completed the previous two: Using the device camera and Recognizing text from an image.

How to do it...

You will now add the barcode reading feature to the existing app:

1. Add the google_mlkit_barcode_scanning package to your project from the Terminal:

   flutter...

Image labeling

ML Kit contains an image labeling service. With it, you can identify common objects, places, animals, products, and more. Currently, the API supports over 400 categories, but you can also use a custom TensorFlow Lite model to add more objects. In this recipe, we’ll learn how to implement this feature in our sample app.

Getting ready

Before following this recipe, you should have completed the Using the device camera and Recognizing text from an image recipes in this chapter.

How to do it...

Let’s now add an image labeling feature to the existing app:

1. Add the google_mlkit_image_labeling package to your project from the Terminal:

   flutter pub add google_mlkit_image_labeling
   

2. In the ml.dart file, import the new package:

   import 'package:google_mlkit_image_labeling/google_mlkit_image_labeling.dart';
   

3. Add a new async method and call it labelImage. The method takes File as a parameter...

Building a face detector and detecting facial gestures

This recipe will explore ML Kit’s face detection. This also includes a model that predicts the probability of a face smiling or having open or closed eyes. This API also includes the identification of key facial features (such as eyes, nose, and mouth) and can get the contours of detected faces.

Getting ready

Before following this recipe, you should have completed the Using the device camera and Recognizing Text from an image recipes in this chapter.

How to do it...

In this recipe, you will add the face detection feature to the existing project: the model will predict whether the faces in the picture are smiling and have their eyes open. Follow these steps:

1. Add the google_mlkit_face_detection package to your project from the Terminal:

   flutter pub add google_mlkit_face_detection
   

2. In the ml.dart file, import the face detection library:

   import 'package:google_mlkit_face_detection...

Identifying a language

Identifying a language can be useful when you get some text information from your users, and you need to respond in the appropriate language.

This is where the ML Kit language package comes to help. By passing some text to the package, you are able to recognize the language of the text. Once you know the language, you can later translate it into one of the other supported languages.

Getting ready

Before following this recipe, you should have completed the Using the device camera and Recognizing text from an image recipes in this chapter.

How to do it...

In this recipe, you will build a screen where users can type some text, and you will identify the language of the text. Follow these steps:

1. Add the latest version of the google_mlkit_language_id package to your project:

   flutter pub add google_mlkit_language_id 
   

2. In the ml.dart file, import the new package:

   import 'package:google_mlkit_language_id...

Using TensorFlow Lite

While using pre-made models to recognize text, objects, and expressions is a powerful and useful feature to add to your apps, there are cases where you need to create your own models or use third-party models that can add virtually limitless features to your projects.

TensorFlow is an open source platform for creating and using ML models, and TensorFlow Lite is a lightweight platform specifically designed to be used on mobile and Internet of Things (IoT) devices.

TensorFlow Hub contains hundreds of models already trained and ready to be used in your own apps. See the https://tfhub.dev/ page for more information.

While creating models and TensorFlow itself are beyond the scope of this book, in this recipe, you will use an open source TensorFlow model built by the creators of the tflite_flutter package.

Getting ready

Before following this recipe, you should have completed the Using the device camera, Recognizing text from an image...

Summary

In this chapter, you explored several powerful features that can be added to your Flutter apps using Google ML Kit.

First, you learned how to use the device camera in your Flutter applications.

Next, you saw how to recognize text from an image using the ML Kit’s Text Recognition API.

You saw how to scan barcodes and QR codes from images: this allows building apps for inventory management, ticket scanning, or any other project that involves reading barcodes.

You used ML Kit’s Image Labeling API to identify objects and scenes in images. This is useful for building apps that can categorize or tag images without any user interaction.

You learned how to use the Face Detection API to identify faces in images and detect facial expressions.

You’ve seen how to identify languages in a text using the Language Identification API and developed a feature that could detect the language of a text input and return its language tag and confidence...