Pulling It All Together for a Complete KM Solution

In the previous three chapters, we have provided a business case for why a knowledge mining (KM) solution will help your organization leverage troves of dark data – or data that sits in the shadows and is unavailable without manual searching by humans. We also discussed the process, in Chapter 5, of building your initial Azure Cognitive Search environment with associated built-in enrichments and indexers for building a metadata repository to enhance the search experience beyond traditional solutions. Finally, in Chapter 6, we provided the process for attaching other Azure Cognitive Services and custom solutions to your Cognitive Search environment to extract maximum detail from your latent data.

In this chapter, we are going to pull it all together to help you understand what a complete KM solution can provide to your organization, with further examples of ways to leverage the services available. To provide these examples...

Technical requirements

To build your complete KM solution with additional Cognitive Services, you will need to have an Azure subscription with at least Contributor rights to the subscription for deploying services. You will need an Azure storage account to store your documents pre- and post-processing. Depending on whether you plan to deploy the solution to a Docker container or an Azure web app, you should have experience with those technologies to ensure proper deployment. You will need to have Visual Studio Code (download Visual Studio Code for macOS, Linux, or Windows at https://code.visualstudio.com/Download) with the node.js extension installed (https://nodejs.org/en/download/package-manager).

Some other services you may use for your deployment are as follows:

• Azure Functions – for building functions and executing code
• Azure Data Factory or similar – for data orchestration and transformation
• Azure Logic Apps – for workflow and triggering...

Getting your Azure environment set up to support your KM solution

As with any deployment to Azure, you should determine what type of project you are building and what type of availability and redundancy you will need to support the project type. This is true even when looking at various production deployments of workloads. Is your application expected to be running all the time such as an e-commerce website, or are you deploying services to perform a once-a-month activity that has some flexibility, such as a movement of on-premises data to archive storage in the cloud? Depending on the business requirements, you will take into consideration how many additional services and capabilities will be required to meet those requirements.

To build your Azure Search solution with custom Cognitive Services and AI enrichments, you will need to understand the business requirements for developing the appropriate solution. In the case of Ocean Smart, our KM solution helps with processing invoice...

Deploying the Azure Cognitive Search service

To support your complete KM solution, you will use Azure Cognitive Search as the foundation for all the additional capabilities you will deploy for maximizing the value of your KM, and many of the features that set apart a KM from traditional data management solutions. For detailed instructions on deploying your Cognitive Search service, refer to Chapter 5. Here, we will focus on specific configurations that will be required when going beyond your proof of concept and looking to deploy a production solution. First, let’s take a look at deploying your storage account.

Azure storage account deployment

As you get started deploying your storage account, keep in mind the application of the solution and the business requirements you need to consider as part of the configuration. If you intend to have multiple search service deployments – for instance, access from multiple geographies – storage redundancy and speed will...

Ingesting your data for indexing and related activities

Ok, so we have discussed how we are going to store the data we will use, but there are many considerations about how we want to attain initial data for indexing and how we will continue to add data to our KM solution to enhance the value of our data.

In common data acquisition scenarios, we can extract the data from a myriad of locations through various Application Programming Interfaces (APIs), databases, document storage locations, and so on. In the case of acquiring and transcribing your audio files, there are a few ways this could be handled, and as the writers of this book, we struggled to determine the best option. The reason for this is somewhat of the positive, yet negative, mailability of the Azure platform. I will spare you some cheesy colloquialism about having many ways of doing the same thing, but know each solution is unique to the organization it is being built for, so oftentimes, it will come down to what is...

Deploying the KM solution with monitoring and notification considerations

The capabilities of the Cognitive Search service discussed in the past few chapters along with the enhanced skills available make it clear how powerful this solution can be. Now, let’s look at a practical deployment of the service for a KM tool within Ocean Smart. We wanted the common user to be able to search for data within the repository of captured files, including documents, images, audio, and video. Using our Django portal that we use throughout the book, we have a simple interface allowing the common user to search and decide which file types they would want to be returned. In the following screenshot, you see an example of a simple search for lobster with the resulting files within a document and image search:

Figure 7.1 – Ocean Smart search interface for identifying files containing information related to our search term

Because of how we have configured the...

Summary

Here, and in the previous few chapters, we have helped to provide you with a broad overview and real solutions to apply the capabilities by combining Azure Cognitive Services technologies. Many organizations struggle with leveraging the significant volumes of data that lie in storage and file shares untapped for the significant value they hold. Beyond simply returning search results and bringing the data to the surface, we can also begin to analyze the data deeper and begin mining the data for other potential uses of AI by building machine learning models to predict future results.

A fully built KM solution is no small undertaking, but the value can greatly outweigh the costs of deployment and services. A small proof of concept to surface some of the details of latent data could be a great place to start, and relatively cost-effective. After making a case for a full implementation, you can harness all the power of the full solution we laid out in these chapters. Whether...