Azure Cognitive Search Overview and Implementation

As we discussed in Chapter 4, Deriving Value from Knowledge Mining Solutions in Azure, the proliferation of data, documents, and other corporate digital assets has perpetuated a need to extract insights from latent data that's left behind. There have been several iterations and attempts at building solutions that can consistently unlock those insights, but all have fallen short and require significant manual intervention. Having the ability to align AI with award-winning accuracy, the flexibility to deploy solutions on-premises, in Azure or other clouds, and a robust and mature search index helps propel Azure Cognitive Search to a category all by itself for your enterprise.

Since you have stayed with us this far in the book and read a fair amount of the overview, history, and operations, it will be a relief to hear that we are going to start building. In this chapter, we will begin the more technical phase of the book, where...

Technical requirements

To build your Azure Cognitive Search solution, you will need to have an Azure subscription with at least Contributor rights to the subscription for deploying services. You will need an Azure Storage account for storing your documents pre- and post-processing. You will need to have Visual Studio Code installed for working with the search indexer. (Download Visual Studio Code for Mac, Linux, and Windows an, https://code.visualstudio.com/Download)

Understanding how Azure Cognitive Search is built

Azure Cognitive Search is a robust and mature service with roots in the Azure Search service that was created back in 2014 originally. The Search service was developed by using the same technology that powered Microsoft Office and Bing Search. Azure Search was retired in 2019 and the capabilities were folded into Azure Cognitive Search with newer, enhanced AI features. The service was originally developed to provide a flexible API-based search solution for developer applications without requiring infrastructure deployments. This freed developers' time up by reducing the number of hours needed to build and maintain an in-house, full-text search solution for their applications.

At its core, Cognitive Search offers two main functions, indexing and querying:

• Indexing – As with other search service providers, indexing is simply the process of loading data and documents into a repository or pointing an indexer to the...

Exploring what services will be used as part of a Cognitive Search solution

Cognitive Search solutions are an amazing way to quickly enrich documents in just about any workload in Azure, on-premises, or in another cloud. You will need to understand, however, the common components required beyond Cognitive Search to deploy a complete solution. In this section, we are going to explore those additional services as Ocean Smart deployed them for the Cognitive Search portion of their KM solution.

For simplicity's sake, all that's really required to deploy the service is a storage account and the Cognitive Search service to have a working solution. After deploying the asset and loading your files into the storage account, you can point your indexer at that storage account and begin an indexing process. This is a fine process for testing the service and understanding how it works, but as you'll see in Chapter 7, Pulling It All Together for a Complete KM Solution, more services...

Pairing common services with Cognitive Search for your KM solution

So, now we know the Cognitive Search service has quite a few capabilities to not only do traditional search activities, but also enrich what we load into the index for extracting additional details that can only be built by reading the documents themselves. We have also looked at what the enhancements can return regarding augmenting the documents and interpreting their contents. However, we have also shown that simply deploying the service does not really help our users search for what they need in a user-friendly interface, which we will need to have for effective use.

This interface may also vary depending on the content of the search index. Because we can use additional Cognitive Services resources to build our complete KM solution, we could potentially load video and use the video indexing service as part of the complete solution, for instance. For these reasons, we are now going to look at the additional services...

Summary

With all the documents related to transactions, purchases, sales, quality, and many more aspects of day-to-day operations at Ocean Smart, many details surrounding these documents can be easily overlooked. Employees are frequently looking for documentation for the purposes of audit retrieval, paperwork related to purchases, and many other uses. Without the ability to extract the additional insights from the documents loaded into the index, it can prove difficult to find these documents if not well organized. Because Ocean Smart is now able to have these additional details in an easy-to-use search interface, finding related paperwork is easier than ever.

This details just one of the many use cases that can be deployed with Cognitive Search. Whether processing incoming payment documents, transcribing and translating recorded audio files to enhance the customer service experience, or indexing video files for speaker and content extraction, Cognitive Search is the foundation...