Experimental Object Recognition with TensorFlow

It’s time for something a bit more experimental. As we’ve seen, artificial intelligence (AI) offers lots of new opportunities to explore when writing code assisted by AI as well as building solutions that are powered by AI. In this chapter, we’ll take a look at TensorFlow. Google developed and published TensorFlow under an open source license. It enables developers to use and train machine learning models for different sorts of applications. You can find a curated list of demos on the TensorFlow website: https://www.tensorflow.org/js/demos.

We’re going to apply a small part of the libraries that Google has published by leveraging the default published model for object recognition.

First, we’ll build a small example prototype to discover some of the capabilities. Then, we’ll apply our newly acquired knowledge to build something experimental and fun. It’s another game, where you have...

Technical requirements

We’ll build the main app on the Vuetify framework (https://vuetifyjs.com/en/) and Pinia (https://pinia.vuejs.org/) to manage the state. As stated previously, we’ll leverage various TensorFlow libraries (https://www.tensorflow.org/js/) to incorporate some intelligence into our app.

You can find the complete code for this chapter here: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/tree/main/09.tensorflow.

Let’s get started with a prototype app!

Introduction to TensorFlow

When I need to research a new framework or technique, I find it very helpful to create a small application for it so that I can test it in complete isolation. We’re going to apply the same approach with TensorFlow. The original idea is that we create an app using the object recognition library (https://github.com/tensorflow/tfjs-models/tree/master/coco-ssd) and apply the model to images from the camera on our device.

Setting up the project

Let’s use a familiar framework to quickly build some boilerplate for our new project. We’ll use the Vuetify CLI to create a new project for us:

1. Run npm create vuetify@3.0.0 in the command-line interface.
2. Choose vue-tensorflow as the project’s name.
3. Use the Essentials (Vuetify, VueRouter, Pinia) installation.
4. Select TypeScript using the arrow keys.
5. Select npm to install the dependencies.

If you navigate to the new project folder, you can run the local development...

Scavenge Hunter

In this section, we’ll build a small app that can run on a web browser, preferably on a mobile phone. With Scavenge Hunter, the goal is to collect certain items from a list. We can use parts of the classes list to control the items our user needs to collect and in that case, we’re sure to be able to detect those objects!

Once an object has been detected, we’re going to add a score based on the find and certainty of the model. Since we can’t guarantee that objects are being recognized properly, we should also be able to skip an assignment. Instead of uploading an image, we’re going to use the camera stream!

Setting up the project

We can continue using the prototype we built or create a new project if we’d like. In the case of the latter, the dependencies and store are required, so we’d need to repeat the relevant steps provided in the Setting up the project and Performing and displaying a status check sections...

Object recognition from the camera

This will be a change that involves a couple of steps. First, we’ll introduce a component that can capture video from the browser. We’ll create a CameraDetect.vue component in the ./components folder: https://github.com/PacktPublishing/Building-Real-world-Web-Applications-with-Vue.js-3/blob/main/09.tensorflow/.notes/9.13-CameraDetect.vue.

The code in the CameraDetect.vue component uses composables from the @vueuse package to interact with the browsers’ Devices and userMedia APIs. We’re using useDevicesList to list the available cameras (lines 33–40) and populate a <v-select /> component (lines 4–14). This allows the user to switch between available cameras.

The user needs to manually activate a camera (also when switching between cameras) for security reasons. The button in the component toggles the camera stream (lines 44–46). To display the stream, we use watchEffect to pipe the stream...

Wrapping up the game flow

Although the game is now functional, it’s not playable since we’re not giving enough feedback to the user. With appStore at our disposal, one of the easiest solutions is to use the dialog! Once we’ve incorporated that, our mini-game will be complete!

First, we’ll update the CameraDetect.vue file by adding the reference to the dialogVisible reactive value. To do this, add the following to the script tag:

// ...abbreviatedimport { useAppStore } from "@/store/app";
const appStore = useAppStore();
const { dialogVisible } = storeToRefs(appStore);
// ...abbreviated

Next, we’ll use dialogVisible in our detectObject function to assess whether it should call the detect function from objectStore:

// ...abbreviatedconst detectObject = async (): Promise<void> => {
  if (!props.disabled && !dialogVisible.value) {
    await detect(video.value, currentCategory.value);...

Summary

We started this chapter with a small prototype to experiment with a new sort of technology. Building something in an isolated environment helps you quickly understand how a certain technology can be adopted in an existing environment. As you’ve experienced, we were running into limitations that could not be resolved. In this case, it didn’t matter that much, since we have few business requirements to deal with.

We also learned how to leverage existing and available APIs from the browser itself to build something unconventional. When putting together a portfolio, standing out with unique projects can make you stand out as a developer. Building little projects while combining multiple technologies can help you understand how you can compose applications with them. This is a more intensive approach but results in a much better understanding of technologies.

Feel free to spend some time customizing projects from previous chapters. In the final chapter, we are...