Streamlining the Quality Control Process with Custom Vision

A seafood company’s reputation can rely heavily on many factors, and Ocean Smart built a reputation for quality by holding higher standards for its products. Customer service, perceived value, quality, and freshness of the product can all be differentiators in a heavily competitive segment of the food industry. The market in the restaurant, food distribution, and retail sectors for frozen lobster tails is quite lucrative. The quality of the products produced is determined by several factors, and when evaluating where AI could be used to help with improving this process, we determined a visual inspection of the product could be assisted by the Custom Vision API. Because the quality control process is so critical, a significant amount of inspection must take place to ensure a quality product is packaged and sent to customers. This inspection is visual – looking at lobster tails moving down the production lines...

Technical requirements

To build your Custom Vision solution, you will need to have an Azure subscription with at least Contributor rights to the subscription for deploying services. You will need an Azure storage account for storing your documents pre- and post-processing. Depending on it, if you plan to deploy the solution to a Docker container or an Azure Web App, you will need experience with those technologies to ensure proper deployment. You will also need to have Visual Studio Code (https://code.visualstudio.com/Download) if you intend to use the Custom Vision SDK. Some other services you may use for your deployment are as follows:

• Azure Functions – for building functions and executing code
• Azure Data Factory or similar – for data orchestration and transformation
• Azure Logic Apps – for workflow and triggering mechanisms

Of course, the more complex and automated your solution is, the more tools you will likely need to know how to use...

Understanding the quality control process at Ocean Smart

The frozen lobster tail market for Ocean Smart is a critical product from a revenue and profit standpoint, and that largely relates to how intricate they are with product inspections. Ensuring that products are inspected properly could mean the difference between a profitable week, month, or quarter, depending on the magnitude of the quality issue and volume of sales. For that reason, there are several areas of the product that are inspected visually and a sample is taken for more close inspection. The quality control team will take a sample of fluid from the product to check for any internal issues with the sample and will then capture and catalog details of the sample. The following is an expectation when an employee is handling the product:

1. It is the role of the employee who is first handling the product to quickly inspect for obvious defects, such as large cracks or obvious blemishes or disease.
2. Throughout the...

Evolving the quality control process with Custom Vision

Computer vision as a discipline has made some amazing strides in recent years to help augment (or enable the augmentation of) human sight. Applications for facial recognition, description of images, and initial sensors being used for self-driving cars have become everyday technology, just to name a few common examples. In the case of self-driving vehicles, computer vision can be used to detect objects in the road, street signs, and the curvature of the road, which then sends signals to the appropriate action by the brakes, accelerator, and so on. However, the origins of computer vision date back to the early 1960s, when a key component of AI systems being developed at universities robots to “see” what was in front of them and respond appropriately. Clearly, the original vision proved to be a bit more challenging than originally anticipated, as those original pioneers would be in their 80s and 90s at their youngest...

Training and improving the model

Many considerations should be taken into account when you are training your model based on the images you are using to build the model. Microsoft recommends at least 50 images for training purposes; however, other details need consideration too. These images should be representative of what a user will face when testing against the model that has been built. For instance, when building the model for frozen lobster tail testing, we had to ensure that tail images were representative of what would be captured by the cameras watching the conveyor belts for tail quality detection. For this reason, the images of the tails needed to capture what would be inspected for quality standards as well. We could not use a clump of lobster tails for our training process, as we couldn’t easily identify the distinct characteristics of a typical tail as it was scanned on the production line.

In simple terms, we had to be very selective about which images to choose...

Using Smart Labeler for faster image labeling

So, now we have reached a point where we have built a model using our images of what a “good” or “bad” frozen lobster tail should look like. As we need to add more images to the model for training and accuracy improvement, we can use the Smart Labeler feature to quicken the process of tagging new images.

Getting started with the process is fairly straightforward now that we already have our model built and a sample set of images loaded. You will simply upload new images within the project using the previous steps, but don’t tag them as you upload them. The images will upload and be labeled as Untagged, with the option to Get suggested tags, as shown in the following figure:

Figure 10.11 – Smart Labeler dialog box displaying Tagged and Untagged image options

When we select the Get suggested tags option, we then get the option to choose how many images we want to tag...

Sending a notification of an irregular product

Our model is built, and we have established a system for capturing images of our tails to compare against our machine learning model. Now, we need to put the downstream processes in place to notify our quality inspector when a Bad Tail has been detected. As described previously, a system for sending notifications and providing feedback when a false positive has been detected was developed using a Raspberry Pi device for input and output of information. If you are unfamiliar with these devices, a Raspberry Pi computer is a lightweight computer commonly running the Linux operating system, which was originally designed for education and hobbyists. Its expansive nature and the add-on components that were designed due to the popularity of the devices led to significantly more adoption of Raspberry Pi devices than originally anticipated, and they have been commonly used in commercial applications as a result.

For our example, we connected...

Building the complete Custom Vision solution

In this chapter, we use Cognitive Services to train a predictive model and deploy the model to a web endpoint to classify new images as they arrive. The following diagram provides an overview of how the tail images are processed and tested against our model to determine whether they are good or bad, as well as how we can frequently update the model to handle new examples:

Figure 10.14 – Process flow of tail image classification testing against machine learning model

To support this process, we create the following:

• The cognitive services and custom vision workspace.
• A storage account where new images and results will be stored and a storage table will be used to communicate results to the frontend web application.
• A function app that contains functions to do the following:
  1. Classify new images as they arrive. It is important to note that we were able to utilize the model trained for the...

Summary

The process for deploying the Custom Vision service itself for smaller examples and PoCs is relatively simple, as described earlier in the chapter. However, to bring the process to operation and establish a process for continual enhancement of the model with a feedback loop is quite a bit more involved, as described in the full deployment outlined in this chapter. As you can see, there are tools for the initial creation that make it easy to get started, but the complete example requires quite a bit of custom code and thought about the best way to provide the feedback necessary to maintain an evolving quality process. This is an important factor when evaluating projects using Cognitive Services as standalone APIs as compared to complete systems and services that can be purchased from vendors. Those vendors could, in fact, be using the APIs provided by Microsoft and training a custom model within the Azure services, but that would just be a part of their application and intellectual...