Operational Analytics

Every business performs certain operations to generate revenue. All these operations eventually generate lots of data, across multiple systems. Many organizations perform operations via the digital space by providing a variety of software applications. All these applications perform a ton of operational logs on the systems they are hosted on. There is a wealth of information in these log files – system errors, performance statistics, security aspects, network traffic patterns, customer information, and so forth. Across all these applications on multiple systems, the amount of daily log data that’s generated can be overwhelming to store, manage, and analyze to get insights from it. Finding relevant pieces of information across all these logs is like finding a needle in a haystack.

To solve this problem of analyzing log data, there needs to be a suitable technology and supporting toolset that can help index all this data and make it easy to search...

Amazon OpenSearch Service

OpenSearch (https://opensearch.org/) is an open source search engine forked from Elasticsearch 7.10.2. It also contains a visualization dashboard component that is forked from Kibana 7.10.2. OpenSearch has a distributed architecture where documents are collected in an index; these indices are divided into shards. Multiple nodes are present in an OpenSearch domain, where shards are replicated across nodes for high availability. In that sense, a lot of configuration, setup, and operational overhead is needed to set up a self-managed OpenSearch platform.

To alleviate the infrastructure setup pain, AWS provides a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters. Amazon OpenSearch Service can either be created in provisioned mode, where users get to select the type and number of nodes to leverage in a cluster, or it can be in serverless mode, where there is no sizing to be done. Provisioned mode is suitable for steady and...

Amazon OpenSearch Service use cases

Let’s dive straight into how Amazon OpenSearch Service can help solve many of the operational analytics use cases that organizations may have.

Application and security monitoring

Use case for application and security monitoring using Amazon OpenSearch Service

GreatFin, being a large financial conglomerate, has many applications, services, systems, and infrastructure setups across all LOBs. The security team wants to leverage a cost-effective solution to provide them with search capabilities in a centralized logging account. This will ensure that security can proactively monitor all threats and look for certain patterns in the log data. At the same time, each LOB also wants to monitor the logs for application issues and latency. The teams are looking for options to ingest data into this operational analytics platform and use dashboards with monitoring and alerting capabilities to look out for specific patterns in the data.

What...

Summary

In this chapter, we looked at what operational analytics is and how organizations want a mechanism to gather insights from all their operational activities. All the system and application log data forms a major component of operational analytics.

We identified OpenSearch as the correct purpose-built tool for storing operational data. OpenSearch has a dashboard component that makes it easy to search for relevant information flexibly and cost-effectively. Along with specific plugins, OpenSearch Dashboards makes it easy to gain insights into log data.

We then looked into how Amazon OpenSearch Service makes the task of managing the OpenSearch cluster easy. Customers can focus on the operational insights part without having to manage the underlying infrastructure alongside the software configurations.

After, we walked through typical use cases that OS helps solve. We looked at a few options to ingest data into the OpenSearch domain. Once the data had been ingested, we looked...

References

• Amazon OpenSearch Service workshop: https://catalog.us-east-1.prod.workshops.aws/workshops/60a6ee4e-e32d-42f5-bd9b-4a2f7c135a72/en-US
• Amazon OpenSearch Service vector database capabilities: https://aws.amazon.com/blogs/big-data/amazon-opensearch-services-vector-database-capabilities-explained/