Chapter 18: X-Pack

X-Pack is a set of modules/plugins that expands the standard capabilities of Elasticsearch, such as the following:

• Security
• Alerting
• Reporting
• Index management
• Monitoring
• Machine learning
• SQL support
• A common extension on mapping, analyzer, search, and ingester processors

It is a commercial component that is managed by the Elastic company, and it's installed by default in Elasticsearch. All the features can be used in trial mode for 30 days, but many of them are free of charge for life in the free tier. The complete map of capabilities offered by different X-Pack tiers is available at https://www.elastic.co/subscriptions.

In this chapter, we will see the most commonly used feature of X-Pack related to everyday usage. For furthermore insights, I suggest checking the official documentation on the Elastic website (https://www.elastic.co). The topics that we will cover in this chapter are as follows:

• ILM –...

ILM – managing the index life cycle

Index Lifecycle Management (ILM) in Elasticsearch is a system that allows you to set up policies to manage different aspects of your index life, such as the following:

• Manage daily, weekly, and monthly indices with backup, optimization, and movement in the different serving tiers (hot and cold)
• Create and switch to new indices when some indices reach a certain size or number of documents
• Delete obsolete indices at the end of their life
• Delete stale indices to enforce retention standards (that is, General Data Protection Regulation (GDPR) and 3-month retention)

ILM policies can be defined via an API or Kibana (go to Stack Management | Index Lifecycle Policies):

Figure 18.1 – The Kibana view of Index Lifecycle Policies

Getting ready

You need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe in Chapter 1...

ILM – automating rollover

ILM is often used in conjunction with rolling-over indices. These indices usually contain append-only data such as logs.

The standard management of the rollover indices is as follows:

• Creation of the index, generally managed by an index template.
• Rolling over the index when it reaches a particular size (the default is 50 GB) or a predefined number of documents.
• If required, move the indices to different tiers.
• Delete the index to be aligned with retention policies.

The rollover functionality is generally used with managed indices called data streams. They are append-only data indices that automatically roll over a new index when it meets some criteria of size and number of documents.

These indices are used mainly for the following:

• Logs
• Events
• Metrics
• Continuously generated data

Getting ready

You need an up-and-running Elasticsearch installation, as we described in the Downloading and installing...

Using the SQL Rest API

X-Pack allows you to bring the power of SQL, the standard language for query data, to Elasticsearch to simplify both the usage of data users and the integration of external applications.

Getting ready

You need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest you use the Kibana console, as it provides code completion and better character escaping for Elasticsearch. We will use the datasets that were populated in Chapter 4, Exploring Search Capabilities and Chapter 7, Aggregations.

How to do it…

We will execute the following steps:

1. We want to return the first five books ordered by quantity; we will use the following API call:

   GET _sql?format=txt
   { "query": "...

Using SQL via JDBC

The SQL access layer for Elasticsearch is able to provide data access via the two most 
used Database Management System (DBMS) protocols, Java Database Connectivity (JDBC) and Open Database Connectivity (ODBC). These drivers can be installed in the client application to easily integrate Elasticsearch with the most popular tools, such as the following:

• DBeaver
• Microsoft Excel
• Microsoft Power BI
• Qlik Sense Desktop
• Tableau

Getting ready

You need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

We will use the datasets that were populated in Chapter 4, Exploring Search Capabilities and Chapter 7, Aggregations. In this recipe, we will configure DBeaver (https://dbeaver.io/) with Elasticsearch; you need to download (https://dbeaver.io/download/) and install it.

A trial license must be active to use a SQL JDBC connection...

Using X-Pack Security

X-Pack Security allows users to secure their production cluster using the security best practices for Elasticsearch.

It covers the following:

• The use of credentials to access an Elasticsearch cluster. Credentials can be a standard user/password stored internally in Elasticsearch or a more complex solution can be used, such as Active Directory and Lightweight Directory Access Protocol (LDAP).
• The use of a Transport Layer Security (TLS)-encrypted connection between the Elasticsearch nodes to prevent traffic sniffing and decoding using custom generated keys.
• The use of SSL endpoints for HTTP calls to secure communication with the cluster.

Getting ready

You need the Docker desktop (https://www.docker.com/) installed on your computer. To be able to execute the Docker commands, you should set up Docker for a production environment; see the Elasticsearch reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html...

Using alerting to monitor data events

Alerting is one of the most used X-Pack components because it allows us to fire alert events on data that is processed in the cluster.

The main concepts behind Elasticsearch alerting are as follows:

• Conditions: These define what needs to be detected.
• Schedule: These define the frequency of how the checks run.
• Actions: These define how to respond to an alert.

Elasticsearch is able to cover the following:

• Infrastructural alerting such as issues about load on the server, disk space, and node being down
• ETL flow alerting such as the reduction of ingested records in some indices
• Business alerting with rules defined by a business user on data quality or features on their data
• Predictive alerting using the Machine Learning (ML) X-Pack component, which is able to detect an anomaly in ingested data

Getting ready

Alerting only works on a full setup environment with security enabled; we will use the...

Why subscribe?

• Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
• Improve your learning with Skill Plans built especially for you
• Get a free eBook or video every month
• Fully searchable for easy access to vital information
• Copy and paste, print, and bookmark content

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [e-mail] for more details.

At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.