In the previous chapter, we learned how to troubleshoot common performance and reliability issues that come up when using Elasticsearch using case studies with real-world examples. This chapter explores some common causes of node and cluster failures. Specific topics covered are as follows:
How to determine the root cause of a failure
How to take corrective action for node failures
Case studies with real-world examples of diagnosing system failures
Elasticsearch node failures can manifest in many different ways. Some of the symptoms of node failures are as follows:
A node crashes during heavy data indexing
Elasticsearch process stops running for an unknown reason
A cluster won't recover from a yellow or red state
Query requests time out
Index requests time out
When a node in your cluster experiences problems such as these, it can be tempting to just restart Elasticsearch or the node itself and move on like nothing happened. However, without addressing the underlying issue, the problem is likely to resurface in the future. If you encounter scenarios such as the ones just listed, check the health of your cluster in the following manner:
Check the cluster health with Elasticsearch-head or Kopf
Check the historical health with Marvel
Check for Nagios alerts
Check Elasticsearch log files
Check system log files
Check the system health using command-line tools
These steps will help diagnose the root cause of problems in your cluster...
This section discusses some real-world scenarios of Elasticsearch node failure and how to address them.
A few weeks ago we noticed in Marvel that the Elasticsearch process was down on one of our nodes. We restarted Elasticsearch on this node, and everything seemed to return to normal. However, checking Marvel later on in the week, we notice that the node is down again. We decide to look at the Elasticsearch log files, but don't notice any exceptions. As we don't see anything in the Elasticsearch log, we suspect that the operating system may have killed Elasticsearch. Checking syslog at /var/log/syslog, we see the error:
Out of memory: Kill process 5969 (java) score 446 or sacrifice child
This verifies that the operating system killed Elasticsearch because the system was running out of memory. We check the Elasticsearch configuration and don't see any issues. This node is configured in the same way as the other nodes in the cluster...
This chapter looked into how to diagnose node failures, determine the root cause of the problem, and apply corrective action. Some key things we learned are:
Many errors, from shard failures to slow query performance, are caused by OutOfMemoryError exceptions
Running out of disk space on one node can cause other nodes to run out of disk space as well when shards are reallocated
Running Elasticsearch alongside other services that require a lot of memory can result in the operating system killing Elasticsearch to free up memory
The next chapter will talk about Elasticsearch 5.0, the next major release of the platform, and it will give you an overview of the various new monitoring tools that will accompany the Elasticsearch 5.0 release.
© 2016 Packt Publishing Limited All Rights Reserved
