Managing Clusters

In the Elasticsearch ecosystem, it's important to monitor nodes and clusters in order to manage and improve their performance and state. There are several issues that can arise at the cluster level, such as the following:

• Node overheads: Some nodes can have too many shards allocated and become a bottleneck for the entire cluster.
• Node shutdown: This can happen due to a number of reasons, for example, full disks, hardware failures, and power problems.
• Shard relocation problems or corruptions: Some shards can't get an online status.
• Shards that are too large: If a shard is too big, then the index performance decreases due to the merging of massive Lucene segments.
• Empty indices and shards: These waste memory and resources; however, because each shard has a lot of active threads, if there are a large number of unused indices and shards, then...

Controlling the cluster health using an API

Elasticsearch provides a convenient way in which to manage the cluster state, and this is one of the first things to check if any problems do occur.

Getting ready

You will need an up-and-running Elasticsearch installation—similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

To execute the commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/) or Postman (https://www.getpostman.com/). You can use the Kibana console as it provides code completion and better character escaping for Elasticsearch.

...

Controlling the cluster state using an API

The previous recipe returns information only about the health of the cluster. If you need more details on your cluster, then you need to query its state.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com/). You can use the Kibana console as it provides code completion and better character escaping for Elasticsearch.

...

Getting cluster node information using an API

The previous recipe allows information to be returned to the cluster level; Elasticsearch provides calls to gather information at the node level. In production clusters, it's very important to monitor nodes using this API in order to detect misconfiguration and any problems relating to different plugins and modules.

Getting ready

You will need an up-and-running Elasticsearch installation—similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com...

Getting node statistics via the API

The node statistics call API is used to collect real-time metrics of your node, such as memory usage, threads usage, the number of indices, and searches.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com/). You can use the Kibana console as it provides code completion and better character escaping for Elasticsearch.

...

Using the task management API

Elasticsearch 5.x and later versions allow you to define the actions that are executed to the server side. These actions can take some time to complete and they can use huge cluster resources. The most common ones are as follows:

• delete_by_query
• update_by_query
• reindex

When these actions are called, they create a server-side task that executes the job; the task management API allows you to control these jobs.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl...

Using the hot threads API

Sometimes, your cluster will slow down due to high levels of CPU usage and you will need to understand why. Elasticsearch provides the ability to monitor hot threads in order to be able to understand where the problem is.

In Java, hot threads are threads that use a lot of CPU and take a long time to execute.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com/). You can use the Kibana console as it provides...

Managing the shard allocation

During normal Elasticsearch usage, it is not generally necessary to change the shard allocation, because the default settings work very well with all standard scenarios. Sometimes, however, due to massive relocation, nodes restarting, or some other cluster issues, it's necessary to monitor or define a custom shard allocation.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com/). You can use...

Monitoring segments with the segment API

Monitoring the index segments means monitoring the health of an index. It contains information about the number of segments and the data that is stored in them.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx.se/) or postman (https://www.getpostman.com/). You can use the Kibana console as it provides code completion and better character escaping for Elasticsearch.

...

Cleaning the cache

During its execution, Elasticsearch caches data in order to speed up searching, such as results, items, and filter results.

Automatically, Elasticsearch frees up memory by following internal indicators such as the percentage size of the cache regarding the memory (that is, 20%). If you want to start some performance tests or free up memory manually, then it's necessary to call the cache API.

Getting ready

You will need an up-and-running Elasticsearch installation – similar to the one that we described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

In order to execute the commands, any HTTP client can be used, such as curl (https://curl.haxx...