Packt+ | Advance your knowledge in tech

You're reading from Learning Elastic Stack 6.0

Product typeBook

Published inDec 2017

PublisherPackt

ISBN-139781787281868

Edition1st Edition

Tools

Kibana Elasticsearch

Concepts

Enterprise Search

Authors (2):

Pranav Shukla

Sharath Kumar M N

View More author details

Chapter 8. Elastic X-Pack

X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, machine learning, and graph capabilities into one easy-to-install package. It adds essential features to make Elastic Stack production ready. Unlike the components of Elastic Stack, which are open source, X-Pack is a commercial offering from Elastic.co, and so it requires a paid license for usage. When you install X-Pack for the first time, you are given a 30-day trial. The basic or free version will provide only monitoring and Dev Tools such as Search Profiler and Grok Debugger. Even though X-Pack comes as a bundle, it allows one to easily enable or disable the features one wants to use.

In this chapter, let's explore the following topics:

Installing X-Pack on Elasticsearch and Kibana
Securing Elasticsearch and Kibana
Monitoring Elasticsearch
Exploring alerting

Installing X-Pack

As X-Pack is an extension of Elastic Stack, prior to installing X-Pack, you need to have both Elasticsearch and Kibana installed. You must run the version of X-Pack that matches the version of Elasticsearch and Kibana.

Installing X-Pack on Elasticsearch

X-Pack is installed just like any plugin to extend Elasticsearch.

These are the steps to install X-Pack in Elasticsearch:

Navigate to the ES_HOME folder.
Install X-Pack using the following command:

$ ES_HOME> bin/elasticsearch-plugin install x-pack

During installation, it will ask you to grant extra permissions to X-Pack, which are required by Watcher to send email alerts and also to enable Elasticsearch to launch the machine learning analytical engine. Specify y to continue the installation or N to abort the installation.

You should get the following logs/prompts during installation:

-> Downloading x-pack from elastic
[=================================================] 100%  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@...

Configuring X-Pack

X-Pack comes bundled with security, alerting, monitoring, reporting, machine learning, and graph capabilities. By default, all of these features are enabled. However, one might not be interested in all the features it provides. One can selectively enable and disable the features that they are interested in from the elasticsearch.yml and kibana.yml configuration files.

Elasticsearch supports the following features and settings in the elasticsearch.yml file:

Feature	Setting	Description
Machine Learning	`xpack.ml.enabled`	Set this to false to disable X-Pack machine learning features
Monitoring	`xpack.monitoring.enabled`	Set this to false to disable Elasticsearch's monitoring features
Security	`xpack.security.enabled`	Set this to false to disable X-Pack security features
Watcher	`xpack.watcher.enabled`	Set this to false to disable Watcher

Kibana supports these features and settings in the kibana.yml file:

Security

Components of Elastic Stack are unsecured, as it doesn't contain inherent security built into it and can be accessed by anyone. This poses a security risk when running Elastic Stack in production. In order to prevent unauthorized access in production, different mechanisms of imposing security such as running Elastic Stack behind a firewall and securing via reverse proxies (such as nginx, HAProxy, and so on) are employed. Elastic.co offers a commercial product to secure Elastic Stack. The offering is part of X-Pack and the module is called Security.

The X-Pack Security module provides the following ways to secure Elastic Stack:

User authentication and User authorization
Node/Client Authentication and Channel Encryption
Auditing

User authentication

User authentication is a process of validating the user and thus preventing unauthorized access to Elastic Cluster. In the X-Pack Security module, the authentication process is handled by one or more authentication services called realms. The...

Monitoring Elasticsearch

Elasticsearch exposes a rich set of APIs known as stats APIs to monitor Elasticsearch at cluster, node, and indices levels. Some of those APIs are _cluster/stats, _nodes/stats, and myindex/stats. These APIs provide state/monitoring information in real time and the statistics presented in these APIs is point-in-time and in .json format. As an administrator/developer, when working with Elasticsearch, one would be interested in both real-time statistics as well as historical statistics, which would help them in understanding/analyzing the behavior (health or performance) of a cluster better.

Also, reading through a set of numbers for a period of time (say, for example, to find out the JVM utilization over time) would be very difficult. Rather, a UI that pictorially represents these numbers as graphs would be very useful in visualizing and analyzing the current and past trends/behaviors (health or performance) of the Elasticsearch cluster. This is where the monitoring...

Alerting

Kibana UI provides beautiful visualizations that help in analyzing and detecting anomalies in data in real time. However, as an administrator or an analyst, it wouldn't be possible to sit in front of dashboards for hours together to detect anomalies and take appropriate action. Wouldn't it be nice if the administrator gets notified when, for example, the following events occur?

There is an outage in one of the servers being monitored
Elasticsearch Cluster turns red/yellow due to some nodes leaving the cluster
Disk space/CPU utilization crosses a specific threshold
There is an intrusion in the network
There are errors reported in the logs

This is where the X-Pack Alerting component comes to the rescue. The X-Pack Alerting component, named Watcher, provides the ability to automatically watch for changes/anomalies in data stored on Elasticsearch and take the required action. X-Pack Alerting is enabled by default as part of the X-Pack default installation.

Watcher provides a set of REST APIs...

Summary

In this chapter, we explored how to install and configure the X-Pack components in Elastic Stack and how to to secure the elastic cluster by creating users and roles. We also learned how to monitor the ElasticSearch server and alerting for generating notifications when there are changes or anomalies in the data.

In the next chapter, we'll put together a complete application using Elastic Stack for sensor data analytics with the concepts learned so far.

The rest of the chapter is locked

You have been reading a chapter from

Learning Elastic Stack 6.0

Published in: Dec 2017Publisher: PacktISBN-13: 9781787281868

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Authors (2)

Pranav Shukla

Pranav Shukla is the founder and CEO of Valens DataLabs, a technologist, husband, and father of two. He is a big data architect and software craftsman who uses JVM-based languages. Pranav has diverse experience of over 14 years in architecting enterprise applications for Fortune 500 companies and start-ups. His core expertise lies in building JVM-based, scalable, reactive, and data-driven applications using Java/Scala, the Hadoop ecosystem, Apache Spark, and NoSQL databases. He is a big data engineering, analytics, and machine learning enthusiast.
Read more about Pranav Shukla

Sharath Kumar M N

Sharath Kumar M N did his master's in computer science at the University of Texas, Dallas, USA. He is currently working as a senior principal architect at Broadcom. Prior to this, he was working as an Elasticsearch solutions architect at Oracle. He has given several tech talks at conferences such as Oracle Code events. Sharath is a certified trainer Elastic Certified Instructor one of the few technology experts in the world who has been certified by Elastic Inc. to deliver their official from the creators of Elastic training. He is also a data science and machine learning enthusiast. In his free time, he likes playing with his lovely niece, Monisha; nephew, Chirayu; and his pet, Milo.
Read more about Sharath Kumar M N

Other recommended products

Related to this chapter

Learning Elastic Stack 7.0

This book teaches you about every component of the Elastic Stack - including Elasticsearch, Kibana, Logstash, and X-pack - with new and the updated features that are released with the 7.0 version. With the help of this book, you will be able to develop enterprise-grade distributed search and analytics applications for your data without any hassle.

BookMay 2019474 pages

Kibana 7 Quick Start Guide

Kibana is the visualization tool of the Elastic Stack, used for visualizing the results of the queries as well the dashboards generated out of the Elasticsearch and Logstash components. This book contains core concepts of Kibana with a straightforward form of chapters so that reader can move forward in a step by step manner.

BookJan 2019172 pages

Mastering Kibana 6.x

Mastering Kibana 6.x provides a rundown explanation required for data visualization and analysis such as X-Pack features, Beats, and machine learning. You will be expert in creating analytics-driven visualizations from a web application. You will be a maestro in creating custom monitoring dashboard using Beats with various examples

BookJul 2018376 pages

Elasticsearch 7 Quick Start Guide

Elasticsearch is one of the most popular tools for distributed search. This book will help you in understanding all about the new features of Elasticsearch 7, and how to use them efficiently for searching, aggregating and indexing data with speed and accuracy.

BookOct 2019186 pages

Mastering Elastic Stack

BookFeb 2017526 pages

Learning Elasticsearch

Elasticsearch is a Lucene-based search and analytics engine for distributed search and analytics. This book will be your hands-on guide as you explore and put to use the features of Elasticsearch 5.x.

BookJun 2017404 pages

Learning Kibana 7

This book will introduce you to Kibana 7, and will show you how it fits into the Elastic stack. You will build a pure metric analytics architecture and visualize it using Timelion. You will also learn how to build relationships between documents using Graph visualization. You will also learn to build powerful Elastic dashboards using Kibana.

BookJul 2019280 pages

Learning Kibana 5.0

BookFeb 2017284 pages

Mastering Elasticsearch 5.x

This book will help you leverage Elasticsearch, guiding you through everything from writing and creating customized plugins to extend Elasticsearch to tackling challenges while handling relational data in Elasticsearch. You’ll learn with the help of practical examples in a step-by-step way.

BookFeb 2017428 pages

Advanced Elasticsearch 7.0

Advanced Elasticsearch 7.0, will help the readers to leverage new features and Core APIs of Elasticsearch to perform advanced search operations. This book covers data modeling, aggregations, pipeline processing, and data Analytics using Elasticsearch

BookAug 2019560 pages

Elasticsearch 5.x Cookbook

BookFeb 2017696 pages

Threat Hunting with Elastic Stack

Elastic security offers enhanced threat hunting capabilities to build active defense strategies. Complete with practical examples and tips, this easy-to-follow guide will help you enhance your security skills by leveraging the Elastic Stack for security monitoring, incident response, intelligence analysis, or threat hunting.

BookJul 2021392 pages

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages

Feature	Setting	Description
Machine learning	`xpack.ml.enabled`	Set to false to disable X-Pack machine...