Chapter 9: Managing Clusters

In the Elasticsearch ecosystem, it's important to monitor nodes and clusters in order to manage and improve their performance and state. Several issues can arise at the cluster level, such as the following:

• Node overheads: Some nodes can have too many shards allocated and become a bottleneck for the entire cluster.
• Node shutdown: This can happen due to a number of reasons, for example, full disks, hardware failures, and power problems.
• Shard relocation problems or corruptions: Some shards can't get an online status.
• Shards that are too large: If a shard is too big, then the index performance decreases due to the merging of massive Lucene segments.
• Empty indices and shards: These waste memory and resources; however, because each shard has a lot of active threads, if there are a large number of unused indices and shards, then the general cluster performance is degraded.

Detecting malfunctioning or poor performance...

Controlling the cluster health using the health API

Elasticsearch provides a health check API to control the cluster state, and this is one of the first things to check whether any problems do occur.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

To execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To control the cluster health, we will perform the following steps:

1. In order to view the cluster health, the HTTP method that we use is GET:

   GET /_cluster/health

The result will be as follows:

{ "cluster_name":"elasticsearch", "status": "yellow",
  "timed_out...

Controlling the cluster state using the API

The previous recipe returns information only about the health of the cluster. If you need more details on your cluster, then you need to query its state.

The cluster state is the core of all functionalities and the metadata stored in Elasticsearch.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To check the cluster state, we will perform the following steps:

1. In order to view the cluster state, the HTTP method that you can use is GET, and the curl command is as follows:

   GET /_cluster/state

2. The result...

Getting cluster node information using the API

The previous recipe allows information to be returned to the cluster level; Elasticsearch provides calls to gather information at the node level. In production clusters, it's very important to monitor nodes using this API in order to detect misconfiguration and any problems relating to different plugins and modules.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To get the cluster node information, we will perform the following steps:

1. To retrieve the node information, the HTTP method you can use...

Getting node statistics using the API

The node statistics call API is used to collect real-time metrics of your node, such as memory usage, threads usage, the number of indices, and searches.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To get the node statistics, we will perform the following steps:

1. To retrieve the node statistics, the HTTP method that we will use is GET, and the command is as follows:

   GET /_nodes/<nodeId1>,<nodeId2>/stats

2. The result will be a long list of all the node statistics. The most significant parts...

Using the task management API

Elasticsearch, from version 5.x, allows you to define actions that are executed on the server side. These actions can take some time to complete, and they can consume huge cluster resources. The most common ones are as follows:

• delete_by_query
• update_by_query
• reindex

When these actions are called, they create a server-side task that executes the job; the task management API allows you to control these jobs.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To get task information, we will perform the...

Using the hot threads API

Sometimes, your cluster will slow down due to high levels of CPU usage, and you will need to understand why. Elasticsearch provides the ability to monitor hot threads in order to be able to understand where the problem is.

In Java, hot threads are threads that use a lot of CPU and take a long time to execute.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

How to do it...

To get the task information, we will perform the following steps:

1. To retrieve the node information, the HTTP method that we use is GET, and the curl command is as follows...

Managing the shard allocation

During normal Elasticsearch usage, it is not generally necessary to change the shard allocation, because the default settings work very well with all standard scenarios. Sometimes, however, due to massive relocation, nodes restarting, or some other cluster issues, it's necessary to monitor or define a custom shard allocation.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index that is populated with the ch04/populate_kibana.sh commands. These are available in the online code.

...

Monitoring segments with the segment API

Monitoring the index segments means monitoring the health of an index. It contains information about the number of segments and the data that is stored in them.

Getting ready

You will need an up-and-running Elasticsearch installation,  similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index that is populated with the ch04/populate_kibana.sh commands. These are available in the online code.

How to do it...

To get information about index segments, we will perform the following steps:

1. To retrieve the index segments, the HTTP method...

Cleaning the cache

During its execution, Elasticsearch caches data in order to speed up aspects of searching, such as results, items, and filter results.

Elasticsearch frees up memory automatically by following internal indicators, such as the percentage size of the cache regarding the memory (that is, 20%). If you want to start some performance tests or free up memory manually, then it's necessary to call the cache API.

Getting ready

You will need an up-and-running Elasticsearch installation, similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console, as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index that is populated with the...