Production and Beyond

This chapter is a flight checklist before going to production. You’ll learn about some important Elasticsearch metrics to monitor once you are in production. Since Elasticsearch is an open source, there are a lot of configurable settings. You’ll learn about some of the most important settings and how to tailor them to your needs. You'll also learn how to install X-Pack and use the monitoring feature of X-Pack.

In this chapter, we will cover the following topics:

• Configuration
• Cluster API
• Monitoring
• X-Pack
• Thread Pools
• Elasticsearch server logs

Configuring Elasticsearch

Elasticsearch is designed to work out of the box. Most settings can be changed on the fly using the settings API. While in development, the default settings should be sufficient. But for production, depending on your application, you need to modify settings, such as memory, file descriptors, and so on. In this section, we will look at the important settings to configure before going to production. First, let's start by looking at the directory structure of Elasticsearch.

The directory structure

You can choose between .zip/.tar.gz or deb/rpm to install Elasticsearch. Depending on how you install, the directory structure, location of log files, the location of the configuration file, and how you...

Multinode cluster

Elasticsearch is a distributed system, and a cluster contains one or more nodes. In this section, we will discuss how to add a new node to the cluster.

Let's say we have two servers with IP addresses 192.168.0.1 and 192.168.0.2.

Follow the instructions laid down next to start a two-node cluster:

1. Install Elasticsearch in the first server. Please follow the instructions in the Installing Elasticsearch section in Chapter 2, Setting up Elasticsearch and Kibana.
2. Open the configuration file in the first server, and make the following changes:

1. 1. Change the cluster name under the cluster section:

                  cluster.name: es-dev

2. 2. Change the node name under the node section:

                  node.name: elasticsearch1

3. 3. Change the host address to bind under the network section:

                  network.host: 192.168.0.1

3. 4. Set the address...

How nodes discover each other

Zen discovery is the discovery module used by Elasticsearch. Since Elasticsearch is a distributed system, you can think of this module as a glue that keeps the cluster together. Cluster management and failure detection are handled automatically by Elasticsearch.

In the configuration file, there is a discovery section dedicated to zen discovery. One of the settings in the discovery section is discovery.zen.ping.unicast.hosts. This setting is a list of other hosts Elasticsearch is running, so that the node can join the existing nodes to form a cluster. When we start the elasticsearch2 instance, this instance will first try to ping the hosts in discovery.zen.ping.unicast.hosts. It will scan the ports 9300 to 9305 and find elasticsearch1 running at 192.168.0.1:9300. (Note that 9300 is the port for internal communication, 9200 is the HTTP server.) To join...

X-Pack

X-Pack is an elastic offering that enables security, alerting, monitoring, and graph functionality for Elasticsearch. X-Pack requires a paid license to use all the features. When you install X-pack for the first time, you are given a 30-day trial. The basic or free version will provide only monitoring. The other features are available with a paid subscription. In this section, we will install X-Pack and discuss on how to update your license to basic if you are not planning to buy a license. For more details on all the subscription offerings of Elasticsearch, please visit the following link:

https://www.elastic.co/subscriptions

X-Pack monitoring, which is part of basic or free license, provides UI with easy-to-read graphs to monitor nodes and the indexes. It is available via Kibana. To use X-Pack, you need both Elasticsearch and Kibana. For instructions on how to install...

Monitoring

Elasticsearch exposes several APIs to monitor cluster health. In this section, we will discuss the native APIs and also X-Pack monitoring, which is available via Kibana. Kibana provides a nice UI and, more importantly, historical data to compare the metrics. You can open Kibana by going to the following URL:

http://localhost:5601/

The default username and password for Kibana are elastic and changeme. You should a screen similar to this:

You can see that monitoring is available for both Elasticsearch and Kibana. If you have Logstash installed, you can also monitor Logstash.

Monitoring Elasticsearch clusters

You can use _cluster API to retrieve the cluster stats:

GET /_cluster/stats/?human&pretty

Refer to the...

Thread pools

Elasticsearch has a thread pool for all the major modules. Each thread pool has a queue associated with the pool. For example, if the index thread pool receives more requests than it can process, the requests are queued up. If the queue is full, the request is rejected. Watching for the number of requests in the queue is important. If the requests are frequently queued, the response times are degraded. The pool size is calculated based on the available processors. The important thread pools are as follows:

Thread pool	Description
index	This is used for index and delete operations. It has a queue size of 200.
search	This is used for search operations. It has a queue size of 1,000.
bulk	This is used for bulk operations. It has a queue size of 50.
refresh	This is used for refresh operations.

Note that if the queue is full, the request is rejected with an...

Elasticsearch server logs

The server logs should be the go-to place when you are trying to figure out why a node is not starting or why shards are not being allocated. The logs provide insight into what's wrong. Elasticsearch uses log4j to handle the logging. The logs are written to the following:

ES_HOME/logs/cluster_name.log

By default, the logs are rotated every day. If you look at the logs directory, you should find something like this:

In the preceding example, the cluster name is es-dev. The current logs are written to es-dev.log. If you want to change the default log level, you can do so using the cluster setting API as shown next. In the following command, we are changing the logging level for the root logger. Elasticsearch also supports changing the log level for a single module-like discovery:

PUT /_cluster/settings
 {
   "transient": {
     "logger...

Summary

In this chapter, we discussed the most important settings to be configured before going to production. You learned about various cluster management APIs to monitor the status of the cluster. We discussed how to use X-Pack monitoring to watch for the important metrics, such as CPU, memory, and so on.

In the next chapter, we will talk about X-Pack premium features, such as graph which can be used to discover relations in your data and alerting which allows you to set up alerts and notifications.