Machine Learning with ESP32

Machine learning (ML) is a huge topic covering different disciplines and knowledge areas. It requires data science, math, algorithm development, signal processing, and subject-matter knowledge about the specific problem that we want to solve. Obviously, our goal in this chapter cannot be to discuss all these underlying subjects, but to learn how to apply ML techniques in IoT projects where possible and beneficial.

Developing an ML application is not an easy task, but developing IoT applications with ML capabilities is extra challenging due to the hardware that we work on. IoT devices are constrained in terms of processing capabilities, memory, and power. As a result, a framework that targets IoT devices has to provide the right tools and libraries that take all these constraints into account. tinyML is the term to describe the field of machine learning that relates to constrained devices.

The topics that we are going to cover in this chapter are...

Technical requirements

The hardware requirements of the chapter are:

• ESP32-S3 Box Lite
• An RGB LED
• A 440 resistor

The libraries and tools to run the examples of the chapter are included in the book repository, therefore there is no need to clone an external repository or download any other third-party tools.

You can find the examples of the chapter here: https://github.com/PacktPublishing/Developing-IoT-Projects-with-ESP32-2nd-edition/tree/main/ch10

Learning the ML basics

Before developing tinyML applications on ESP32, we will talk about some ML basics and terminology that will help us to better understand the examples of the chapter and ML concepts in general. As alluded to in the introduction, ML is not for all types of computing problems. We will discuss how ML differs from traditional programming and the types of ML approaches to solve different classes of problems. Then we will have an overview of the tinyML pipeline to understand the overall process of developing an ML application on a constraint device.

Let’s start with what ML is and what we can do with it.

ML approaches to solve computing problems

In traditional software development, we apply a set of rules (the program) to data (input) to get a result (output). Since we know the requirements, we write a specific program to generate a distinct output from data. We can see the flow in traditional programming in the following figure:

Figure...

Running inference on ESP32

Deep learning is a supervised learning method in ML. Similar to the human brain, it contains neurons, ie. computational units. Neural networks (NNs) are implementations of the deep learning method. A neural network has several layers of neurons and each layer can have a different number of neurons. In the last layer, the neural network generates its prediction. There are different types of layers, such as a fully connected layer, convolutional layer, pooling layer, etc. In a fully connected layer, all neurons are connected to every neuron in the next layer so that they can pass their calculations to the next layer fully. The following figure shows an NN with fully connected layers:

Figure 10.3: A fully-connected NN (Source: Wikimedia Commons)

An NN utilizes number arrays, or tensors, as a data structure. For example, a vector is a One-Dimensional (1D) tensor, and a matrix is a 2D tensor. A scalar, a single number, is a 0D tensor. Data is represented...

Developing a speech recognition application

Speech recognition is another field where ML algorithms can do a very good job and Espressif invests a lot in this field in terms of both hardware and software. The ESP32-S3 series of MCUs is a response to the advancements in ML. When we look at the ESP32-S3 technical reference manual, we see that ESP32-S3 has an extended instruction set to support 128-bit vector operations with an additional eight 128-bit general-purpose registers. Having a Single Instruction, Multiple Data (SIMD) paradigm in mind, the ESP32-S3 Arithmetic Logic Unit (ALU) is capable of, for example, processing 16 8-bit vectors in a single instruction with this extended instruction set, or Processor Instruction Extensions (PIEs), as they’re called in the ESP32-S3 technical reference.

Espressif supports this hardware with advanced frameworks and libraries. Here is a short summary of them:

• ESP-DSP: The Digital Signal Processing (DSP) library for vector...

Summary

Advances in MCUs and AI/ML algorithms have made it possible to have models small enough to be run on IoT devices. tinyML enables us to develop ML applications that can make inferences on data coming from sensors right in the field without the need for a round-trip to a backend system or cloud. There are several frameworks that we can employ for this purpose. TensorFlow is a platform that provides all the necessary tools and libraries to develop ML applications.

Data collection, ML model design and optimization, and inference are the stages in the tinyML pipeline. TensorFlow Lite for Microcontrollers (TFLM) is the framework in TensorFlow to optimize and use ML models on constrained IoT devices. We developed a simple tinyML application that uses a sine-wave model to make inferences and draws its predictions on the screen. Espressif empowers us with two models to develop voice applications: WakeNet and MultiNet. We learned how to use these models in an application that activates...

Questions

Here are some questions to review what we have learned in the chapter:

1. Which one of the following is not true about machine learning?
   1. Supervised learning needs labeled data to train models.
   2. Unsupervised learning tries to find outliers in data.
   3. The agent in reinforced learning interacts with the environment to learn.
   4. Reinforced learning is superior to others at detecting patterns in data.
2. Which one of the following is not a step in the tinyML pipeline?
   1. Data collection and pre-processing
   2. Training the model on an IoT device
   3. Optimizing the model for deployment
   4. Running inference on an IoT device
3. Which technique makes an ML model small enough to fit into the memory of an IoT device?
   1. Training
   2. Quantization
   3. Overfitting
   4. Validation
4. With TFLM, we can:
   1. Optimize a TensorFlow model...

Further reading

ML is a huge subject to learn about. The following books can be helpful on this journey:

• TinyML Cookbook, Gian Marco Iodice, Packt Publishing (https://www.packtpub.com/product/tinyml-cookbook/9781801814973): Discusses tinyML on Arduino and Raspberry Pi with examples. Chapter 3 shows the complete ML pipeline in an example, starting from preparing a dataset, to training a model, quantization, and inference.
• Deep Learning with TensorFlow and Keras, Amita Kapoor, Antonio Gulli, Sujit Pal (https://www.packtpub.com/product/deep-learning-with-tensorflow-and-keras-third-edition/9781803232911): Explains deep learning. A must-read book to design effective NN models.

Learn more on Discord

To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:

https://discord.gg/3Q9egBjWVZ