Chapter 9. X-Pack: Security and Monitoring

In previous chapters, we have explored four core components of Elastic Stack - Elasticsearch, Logstash, Kibana and Beats. While these components help us with pipelining the data, indexing and visualization, there are still few aspects left which are even more important when it comes to production level setup. Security comes to mind when we talk about a server in production. We would want to be notified for events such as errors, faults and so on. Such important features are at core of X-Pack. 

In this chapter, we will explore the various components present in X-Pack in brief. We will cover the need for X-Pack followed by the features, installation, and configuration of the components present within X-Pack. 

In this chapter, we will cover the following topics:

• Introduction to X-Pack

• Installation of X-Pack

• Exploring security

• Viewing X-Pack information

• Exploring monitoring

• Understanding Profiler

Introduction to X-Pack

Before learning about X-Pack, let's understand how X-Pack came into existence. We have discussed the need for Elastic Stack, where earlier there were different versions maintained for Elasticsearch, Logstash, Kibana, and Beats. Therefore, to avoid confusion for the users, Elastic Team. simplified and came up with Elastic Stack where there would be a single release version for Elasticsearch, Logstash, Kibana, and Beats. After the rise in use of these products, there was a need for supporting products that would help the end user such as a need for authorization and authentication of Elasticsearch clusters and Kibana, a need for monitoring the Elasticsearch cluster using a simple yet intuitive UI, and a need for robust alerting and notification mechanisms.

X-Pack is an extension package for Elastic Stack, which combines the various product offerings apart from the components of Elastic Stack, such as Shield, Marvel, Watcher, and Graph, and provides additional features...

Installation of X-Pack

To install X-Pack, you need to have installed Elasticsearch 5.0 and Kibana 5.0 onwards. Also, you must run the version of X-Pack that matches the version of Elastic Stack that you are running.

For the installation of Elasticsearch, you can refer to the Installation of Elasticsearch section in Chapter 1 , Overview of the Stack. For the installation of Kibana, you can refer to the Installation of Kibana section in Chapter 1 , Overview of the Stack. X-Pack will be installed in both Elasticsearch and Kibana, for which the Elasticsearch plugin and Kibana plugin will be used.

Installing X-Pack in Elasticsearch

In order to install X-Pack in Elasticsearch, refer to the following steps:

1. Install X-Pack while running the following command from the Elasticsearch installation directory - bin/elasticsearch-plugin install x-pack.

   Note

   Run the installation as a superuser if you are using DEB/RPM distributions.

   During installation, it will ask you to grant extra permissions to X-Pack, which...

Security

X-Pack security is a module that has been created out of the requirement of having a proper authorization mechanism to access the data present in the cluster. It initially started with the need for secure the Elasticsearch cluster, and it has grown beyond that to even secure the Kibana UI and provide access to authorized users only.

Whenever we talk about security, we tend to discuss the three As:

• Authentication: Used to authenticate users based on their identity

• Authorization: Describes the roles/permissions granted to an authenticated user

• Accountability: Logging user session information, usage information, and so on

Considering the three As, Elastic Team is using Shield as an X-Pack Security model, which makes sure that the 3A's are covered.

For Authentication, it provides restrictions on unauthorized access with basic password protection, organization-level user management, and IP-based authorization.

For Authorization, it provides role-based access control, which defines what roles...

Viewing X-Pack information

X-Pack provides an API which we can use to view its information. It provides us with the build details, license details, and the details of each component of X-Pack.

To view the information using console in the Kibana UI, the command will be as follows:

GET /_xpack

It will give us the following response:

{ 
  "build": { 
    "hash": "821d294", 
    "date": "2016-12-06T13:09:18.057Z" 
  }, 
  "license": { 
    "uid": "e065e495-3fb4-4cb5-8233-263074ee57e7", 
    "type": "trial", 
    "mode": "trial", 
    "status": "active", 
    "expiry_date_in_millis": 1485882488285 
  }, 
  "features": { 
    "graph": { 
      "description": "Graph Data Exploration for the Elastic Stack", 
      "available": true, 
      "enabled": true 
    }, 
    "monitoring": { 
      "description": "Monitoring for the Elastic Stack", 
      "available": true, 
      "enabled": true 
...

Monitoring

Monitoring is another X-Pack component that has grown out of the requirement of having a UI to monitor the cluster, indices, and nodes present in Elasticsearch. It provides us with detailed statistics for each cluster, each of the indices present in the cluster, and for each of the nodes.

By making Monitoring incorporated in X-Pack, now you can access the Monitoring UI directly from Kibana, thereby eliminating the need to leave Kibana. Earlier, it was known as Marvel, which provided you with a UI to monitor and get statistics of clusters, nodes, and indices present in Elasticsearch. You can even see the performance of the Kibana instance that is running, which is an addition.

Monitoring consists of two sub-components: a monitoring agent and a monitoring application. The monitoring agent is to be installed on every node from which you want to fetch statistics, and collects the data and indices of the data in Elasticsearch on which data is visualized using the Monitoring dashboards...

Understanding Profiler

Profiler is another X-Pack component which has grown out of requirement of understanding how a request performs in Elasticsearch. It provides us with detailed information of how the request has been executed at various stages of the request as sent which can be useful to find why your requests are being served slowly or what causes the requests to return results after a long time.

It is the latest addition to X-Pack and has been introduced in X-Pack version 5.1 itself. It is part of the Dev Tools Page. It utilizes the Profile API which is provided by Elasticsearch. Profile API is used to debug the various information of a request to gather underlying details of how request was served and how result was received. The resulting information of Profile API is a large chunk of JSON which is difficult to understand and make sense of. Therefore, Profiler plays an important part of providing the results of Profile API in form of a visualization which makes it easier to understand...

Summary

The Elastic Team kept core components in Elastic Stack and supporting features separately as X-Pack, which is an important asset for any organization. In this chapter, we learned about two of the X-Pack team members: Security and Monitoring. We got to set up X-Pack with Elasticsearch and Kibana. Under security, we learned to manage users, roles, privileges, and so on, while Monitoring helped us to look at different visualizations showcasing the real-time statistics of Elasticsearch and Kibana instances.

In the next chapter, we will cover the rest of the components of X-Pack: alerts, graphs, and reporting.