You're reading from Modern Data Architecture on AWS

Product typeBook

Published inAug 2023

PublisherPackt

ISBN-139781801813396

Edition1st Edition

Concepts

Data Science

Author (1)

Behram Irani

Operational Analytics

Every business performs certain operations to generate revenue. All these operations eventually generate lots of data, across multiple systems. Many organizations perform operations via the digital space by providing a variety of software applications. All these applications perform a ton of operational logs on the systems they are hosted on. There is a wealth of information in these log files – system errors, performance statistics, security aspects, network traffic patterns, customer information, and so forth. Across all these applications on multiple systems, the amount of daily log data that’s generated can be overwhelming to store, manage, and analyze to get insights from it. Finding relevant pieces of information across all these logs is like finding a needle in a haystack.

To solve this problem of analyzing log data, there needs to be a suitable technology and supporting toolset that can help index all this data and make it easy to search...

Amazon OpenSearch Service

OpenSearch (https://opensearch.org/) is an open source search engine forked from Elasticsearch 7.10.2. It also contains a visualization dashboard component that is forked from Kibana 7.10.2. OpenSearch has a distributed architecture where documents are collected in an index; these indices are divided into shards. Multiple nodes are present in an OpenSearch domain, where shards are replicated across nodes for high availability. In that sense, a lot of configuration, setup, and operational overhead is needed to set up a self-managed OpenSearch platform.

To alleviate the infrastructure setup pain, AWS provides a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters. Amazon OpenSearch Service can either be created in provisioned mode, where users get to select the type and number of nodes to leverage in a cluster, or it can be in serverless mode, where there is no sizing to be done. Provisioned mode is suitable for steady and...

Amazon OpenSearch Service use cases

Let’s dive straight into how Amazon OpenSearch Service can help solve many of the operational analytics use cases that organizations may have.

Application and security monitoring

Use case for application and security monitoring using Amazon OpenSearch Service

GreatFin, being a large financial conglomerate, has many applications, services, systems, and infrastructure setups across all LOBs. The security team wants to leverage a cost-effective solution to provide them with search capabilities in a centralized logging account. This will ensure that security can proactively monitor all threats and look for certain patterns in the log data. At the same time, each LOB also wants to monitor the logs for application issues and latency. The teams are looking for options to ingest data into this operational analytics platform and use dashboards with monitoring and alerting capabilities to look out for specific patterns in the data.

What...

Summary

In this chapter, we looked at what operational analytics is and how organizations want a mechanism to gather insights from all their operational activities. All the system and application log data forms a major component of operational analytics.

We identified OpenSearch as the correct purpose-built tool for storing operational data. OpenSearch has a dashboard component that makes it easy to search for relevant information flexibly and cost-effectively. Along with specific plugins, OpenSearch Dashboards makes it easy to gain insights into log data.

We then looked into how Amazon OpenSearch Service makes the task of managing the OpenSearch cluster easy. Customers can focus on the operational insights part without having to manage the underlying infrastructure alongside the software configurations.

After, we walked through typical use cases that OS helps solve. We looked at a few options to ingest data into the OpenSearch domain. Once the data had been ingested, we looked...

References

Amazon OpenSearch Service workshop: https://catalog.us-east-1.prod.workshops.aws/workshops/60a6ee4e-e32d-42f5-bd9b-4a2f7c135a72/en-US
Amazon OpenSearch Service vector database capabilities: https://aws.amazon.com/blogs/big-data/amazon-opensearch-services-vector-database-capabilities-explained/

The rest of the chapter is locked

You have been reading a chapter from

Modern Data Architecture on AWS

Published in: Aug 2023Publisher: PacktISBN-13: 9781801813396

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Author (1)

Behram Irani

Behram Irani is currently a technology leader with Amazon Web Services (AWS) specializing in data, analytics and AI/ML. He has spent over 18 years in the tech industry helping organizations, from start-ups to large-scale enterprises, modernize their data platforms. In the last 6 years working at AWS, Behram has been a thought leader in the data, analytics and AI/ML space; publishing multiple papers and leading the digital transformation efforts for many organizations across the globe. Behram has completed his Bachelor of Engineering in Computer Science from the University of Pune and has an MBA degree from the University of Florida.
Read more about Behram Irani

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages