Project – Baby Monitor

Audio analysis and classification is one of the fields where machine learning (ML) yields very successful results. In this final chapter of the book, we are going to develop an ML application on ESP32-S3 to detect baby cries in the environment. Edge Impulse will be the ML platform of the project since it provides all the necessary tools and functionality that we need for this project. The application will also connect to the RainMaker platform and pass information to the cloud to notify users via the RainMaker mobile application. A typical Internet of Things (IoT) solution usually requires one or more third-party systems/platforms to be integrated into the same solution. This project provides us with a good opportunity to gain hands-on experience with several integrations and see what problems can arise during development.

In this chapter, we will discuss the following topics to develop a baby monitoring solution:

• The feature list of the...

Technical requirements

The only hardware requirement for this chapter is the ESP32-S3 Box Lite. We will develop the entire project on this devkit without any additional sensor hardware since the devkit already has two microphones integrated into the box.

On the software side, we will use mobile applications and online platforms to implement the project. They are:

• ESP RainMaker platform: The cloud platform of the project.
• ESP RainMaker mobile application: The companion application that comes with the RainMaker platform to add devices and manage them in the platform. It is available for both Android and iOS.
• Amazon Alexa: The mobile application by Amazon to control and monitor Alexa-enabled devices. It is available for both Android and iOS.
• Edge Impulse: The platform to generate the ML model for detecting baby cries.

You can find the project codes in the GitHub repository here: https://github.com/PacktPublishing/Developing-IoT-Projects-with...

The feature list of the baby monitor

The baby monitor device will have the following features:

• It will detect when a baby cries.
• It will define a node on ESP RainMaker.
• When a baby-crying event is detected, it will be reported to ESP RainMaker.
• The mobile application will show a notification on the mobile device for the event.
• It will be integrated with Amazon Alexa to provide input to Alexa routines.

The project may only have a very brief feature list; nonetheless, it introduces some interesting challenges. Let’s begin with discussing the solution architecture in the next topic.

Solution architecture

As discussed, the goal of the project is to develop a smart device to detect baby cries. When it detects the event, it will pass this information to the ESP RainMaker platform and generate a notification on the mobile application. The solution will also integrate with Amazon Alexa and baby-crying events will be reflected on Amazon Alexa as well. The following figure shows this basic architecture:

Figure 12.1: ESP RainMaker Solution components

The ESP RainMaker platform (https://rainmaker.espressif.com/) will provide the following functionality:

• Device provisioning – adding/removing devices via the mobile application
• Remote monitoring
• Mobile notifications
• Alexa Voice Service integration

The other platform that we are going to employ in this project is Edge Impulse. We need to have an ML model to analyze audio data and make inferences to decide whether a baby’s cry is detected. We don’t need...

Implementation

The implementation will have only one application running on the ESP32-S3 Box Lite. It sounds relatively easy to develop, but each project comes with its own challenges. The challenge of this project is memory management. We will discuss memory management and optimization in detail throughout the implementation. The first step is to generate the ML model.

Generating the ML model

As we mentioned earlier, we will find a baby-crying detection model from the Edge Impulse project repository and download the C++ SDK with the model in it. The following steps show how to do that:

1. Go to https://edgeimpulse.com/projects/ and type baby-cry-detector in the search box. Select the first project from the results:

   Figure 12.3: Baby-cry-detector project on Edge Impulse

2. Clone the project by pressing the Clone this project button at the top right:

   Figure 12.4: Cloning the Edge Impulse project

3. Log in to your Edge Impulse...

Testing the project

In the test, we will connect the devkit to the RainMaker platform, as we did earlier in Chapter 7, ESP32 Security Features For Production-Grade Devices, then provide baby-crying sounds to see whether the event is generated and passed to the RainMaker platform. Let’s do this in steps:

1. Make sure the devkit is not registered as another device in the RainMaker platform by checking the RainMaker mobile application. If registered, remove it from the mobile application.

   Figure 12.16: The RainMaker mobile application with no device registered

2. Flash the application on the devkit:

   $ idf.py erase-flash flash monitor
   

3. A QR code will be printed on the serial console. Scan it with the RainMaker mobile application by pressing the Add Device button that is shown in step 1.

   

   Figure 12.17: QR code to provision the device to the local WiFi, then the RainMaker

4. Confirm that the device is provisioned...

Troubleshooting

There can be several pitfalls in this project. The following list can help you in such cases:

• Integration is probably the most difficult part of the project if you decide to configure it yourself. There are several frameworks (ESP-RainMaker, ESP-SkaiNet, Edge Impulse SDK, and BSP of the devkit) that need to work together on the ESP32-S3 Box Lite. Make sure each of them compiles on your development machine and works on the devkit separately before the integration. You can create different IDF projects for this purpose.
• As we mentioned at the beginning of the project, the sdkconfig file keeps key configuration items. Use the menuconfig tool (idf.py menuconfig) to check whether sdkconfig contains the correct configuration values.
• While integrating with the ESP RainMaker platform, make sure everything is in a clean state – that is, there is no other device provisioned on RainMaker. You can also erase the flash memory of the devkit and use...

New features

The project has a lot of room to improve. Here are some ideas for practice:

• RainMaker OTA update is not enabled on the device. It is especially critical for ML applications to update the device firmware with the new version of the improved model. You can enable OTA update in the application and try upgrading device firmware over the RainMaker platform.
• You can develop another RainMaker node application on ESP32-C3-DevKitM-1 as a companion device that can turn its LED on when a baby-crying event is detected. RainMaker automations would be an easy way to make the devices work together.
• There is an LCD on the ESP32-S3 Box Lite. You can add some graphics features to the application by using LVGL and SquareLine Studio, such as displaying provisioning QR codes on the screen, RainMaker connection status, and baby-crying events in operation.
• RainMaker supports Alexa Voice Service (AVS) integration. Nonetheless, the use of the device on AVS is limited...

Summary

Developing an IoT product usually means a lot of integration with third-party systems and platforms. In this last chapter of the book, we had an example of this by developing a connected machine-learning application on ESP32-S3. We downloaded an ML model from the Edge Impulse platform and updated the Edge Impulse SDK for our devkit. On the cloud side, we employed ESP RainMaker. We defined a RainMaker node, device, and parameter in the application to exchange data with the RainMaker platform. The challenge of the project was memory usage. The internal memory of ESP32-S3 was not enough to accommodate all the functionality as specified in the requirements; therefore, we enabled and used SPIRAM to keep the buffers on it. When we look at real-world IoT projects, they also usually need to address such issues, and we had a hands-on experience by working on the project of this chapter.

For the next steps, I recommend you try other chips from Espressif Systems. Espressif Systems...

Chapter 02:

1. CMakeLists.txt. ESP-IDF integrates cmake as its build system.
2. sdkconfig. menuconfig is the tool to edit the project configuration parameters in sdkconfig.
3. idf.py. This Python tool comes with ESP-IDF installation and is used to manage ESP-IDF projects.
4. Built-in JTAG/USB. ESP32-S3 doesn’t require an extra hardware tool for JTAG access since it has a built-in JTAG circuitry.
5. platformio.ini. This file is added with each new PlatformIO project and defines the project configuration.

Chapter 03:

1. An LED needs a GPIO connection to drive as a digital output.
2. I²C requires two lines, clock and data, to communicate with other devices. The protocol supports multiple slaves on the same bus with addressing.
3. Higher transfer rate is the reason for choosing SPI communication for SD cards.
4. WS selects the right or left channel in I2S communication.
5. MRI is not a display technology for IoT.

Chapter 04:

1. FlatBuffers is especially useful when we need to pass data between different platforms.
2. Miniz is a library to compress data – it works especially well on data with repeating patterns.
3. LVGL is a graphics library, but it doesn’t include display drivers.
4. The ESP-IDF Components library is not provided by Espressif Systems.
5. ESP-ADF is the framework for audio processing.

Chapter 06:

1. In STA mode, ESP32 can also retrieve a dynamic IP from the Wi-Fi router it connects to and use it to communicate on the network.
2. Although Near-Field Communication (NFC) can be used as a method for provisioning, it is not supported by ESP-IDF directly.
3. MQTT defines topics to publish and subscribe
4. HTTP GET is the method to request a resource
5. All of the above 2XX codes show a successful request, 4XX codes mean that the client didn’t send an expected request to the server, and 5XX codes are returned when an error occurs while processing the request

Chapter 07:

1. Flash encryption. When it is enabled, the firmware on the flash cannot be reverse-engineered.
2. A secure channel is not required for OTA updates, but it creates a security hole to be exploited by intercepting the network traffic to obtain the new firmware. The best practice is to do OTA updates over a secure channel.
3. The RainMaker platform does require mutual authentication with devices.
4. ESP Insights is a tool to monitor ESP32 devices in the field remotely.
5. All of them are important. Each component in an IoT solution creates an attack surface for hackers.

Chapter 08:

1. AWS IoT Device SDK helps us to connect ESP32 to the AWS cloud.
2. IoT rules are used to pass messages to the other AWS services from IoT Core.
3. AWS provides Grafana as a managed service, so there is no need to install Grafana on an EC2 instance.
4. Check the lambda handler first. It is easy to test and provides logs.
5. Using the lambda test tool and sending a discovery message to the lambda function would lead to the solution faster.

Chapter 10:

1. Each category of ML algorithms responds to different classes of problems in nature and they are not comparable in terms of superiority at detecting patterns.
2. Training an ML model requires computational resources that are not available on IoT devices.
3. Quantization is a technique to shrink ML models to fit into IoT devices.
4. TFLM comes with the tools to optimize ML models for IoT devices.
5. MultiNet is the model by Espressif for detecting voice commands offline.

Chapter 11:

1. MLOps platforms help with data and model management and sometimes come with an integrated AutoML tool, but this is not always the case.
2. Edge Impulse doesn’t connect to remote databases to retrieve data.
3. With transfer learning, we can add new layers on top of the source model to fit our purposes.
4. ESP-NN provides the functionality for neural network operations.
5. ONNX is an open format for ML models.