Chapter 13: Architecting Workloads on the Elastic Stack

Over the last 12 chapters, this book has primarily focused on getting started with Elastic Stack. We looked at each core component of Elastic Stack in detail and understood how they all come together to solve a range of data-related use cases. We also looked at how Elastic Stack can be leveraged to solve turnkey solutions in search, observability, and security use cases.

The focus of this book was on giving you the best possible introduction and starting point for your journey on the stack. We also spent time understanding and contextualizing the domains that we were designing our solutions for. The final aspect of getting started with Elastic Stack is understanding how to architect and design your workloads for long-term success; it is important to ensure the solution you put in place evolves with your use cases, is easy to operate and maintain, and is efficient and fit for purpose, given the nature of the workload.

This...

Architecting workloads on Elastic Stack

As we discussed in the first few chapters of this book, Elastic Stack consists of several fundamental components that work together to handle your big data workloads and solve use cases in the different solution areas. Given each component is run independently, there is a great deal of flexibility in how the stack is deployed and configured. The architecture for your solution comes down to the requirements or constraints you need to consider for your environment.

When you think about the architecture for Elastic Stack, it makes sense to categorize the core components into layers, as shown in the following diagram:

Figure 13.1 – Elastic Stack components in layers

Solution architects generally consider several key architecture principles or best practices when putting together the design for a mission-critical platform. As with most things in IT, not all of these are black and white requirements, and it is important...

Architectures to handle complex requirements

Elastic Stack is designed to be extremely flexible in how it can be deployed and used to solve your big data use cases. This section will explore specialized architecture patterns that can be used to handle more complex architectural requirements for your solution.

Elasticsearch (and other stack components) can be configured to be highly available and fault-tolerant in a single geographical region when following standard architecture best practices. Given the sensitive latency and bandwidth requirements for inter-node communication within an Elasticsearch cluster, it is generally not possible to deploy a cluster that spans multiple geographical regions. This presents two main challenges:

• Standard architecture clusters are not resilient to complete failures within a geographical region.
• Users and data in other geographies will experience high latencies, pay more in terms of data transfer costs, and have reduced bandwidth for...

Implementing successful deployments of the Elastic Stack

Once you have designed and implemented the right architecture for your deployment of Elastic Stack, it is important to consider some of the operational best practices when it comes to running and maintaining your deployment.

While fundamental considerations for running large-scale IT platforms such as patching/vulnerability management, privileged access management, quality of service/quota management, and others apply, this section focuses on some quick and easy wins to improve operational resiliency and processes for you to consider as you get started with your Elastic Stack journey:

• Monitoring and observing Elastic Stack components

It is important to consider how critical components of Elastic Stack will be monitored and observed as part of your broader IT environment. Elastic Beats Agents come with out-of-the-box modules for Elasticsearch, Kibana, Logstash, and Beats themselves to collect logs and metrics...

Summary

In this chapter, we looked at how Elastic Stack components can be architected and deployed to satisfy commonly expected nonfunctional requirements in modern IT environments. First, we looked at various design considerations when architecting workloads for high availability, scalability, disaster recoverability, and security. We understood the characteristics of the different Elastic Stack components and how they can be deployed to achieve these requirements.

Next, we looked at more complex Elastic Stack architectures to handle specialized requirements and use cases. We looked at how cross-cluster search architectures enable federated searching across Elastic Stack deployments, potentially in different geographical regions. We also looked at how cross-cluster replication can be used to enable multi-region DR use cases while improving speed and access to data for globally dispersed users. We then looked at using tiered data architectures on Elasticsearch to help teams make...

Why subscribe?

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
• Improve your learning with Skill Plans built especially for you
• Get a free eBook or video every month
• Fully searchable for easy access to vital information
• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [e-mail] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.