In this chapter, we will look at the role of Elasticsearch in ELK Stack. It covers the features of Elasticsearch, and why it is such a wonderful technology to enable fast search responses for real time analytics. In the end, we will also briefly look at some of the plugins available for Elasticsearch, which make our lives much easier while dealing with the Elasticsearch cluster.
Elasticsearch is a search and analytics engine that enables fast and scalable searches in a distributed environment. As we have already covered in Chapter 1, Introduction to ELK Stack, some of the biggest distributed architectures, such as GitHub, StackOverflow, and Wikipedia, make use of the Elasticsearch full-text search, structured search, and analytics capabilities for fast and relevant searches.
Elasticsearch is built on Apache Lucene. The definition of Lucene from its Apache page (https://lucene.apache.org) is:
"Apache LuceneTM is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform"
Elasticsearch hides the complexity behind Lucene by providing a powerful RESTful API built on top of it, which makes querying the indexed data easier, and makes it available to any programming language. It extends the capabilities of Lucene by...
Let's look at some of the basic concepts of Elasticsearch, which explain how it stores the indexed data.
Index in Elasticsearch is a collection of documents that share some common characteristics.
Each index contains multiple types, which in turn contains multiple documents, and each document contains multiple fields. An index consists of multiple JSON documents in Elasticsearch. There can be any number of indices in a cluster in Elasticsearch.
In ELK, when Logstash JSON documents are sent to Elasticsearch, they are sent as the default index pattern "logstash-%{+YYYY.MM.dd}". It partitions indices by day so that it can easily be searched and deleted if required. This pattern can be changed in the Logstash output plugin configuration.
The URL to search and query the indices looks like this:
http://localhost:9200/[index]/[type]/[operation]
In ELK, although Logstash and Kibana act as an interface to talk to Elasticsearch indices, it's still necessary to understand how Logstash and Kibana makes use of Elasticsearch RESTful APIs to perform various operations, such as creating and managing indices, storing and retrieving the documents, and forming various types of search queries around the indices. It is also often useful to know how to delete indices.
As we already know, Elasticsearch provides an extensive API to perform various operations. The generic syntax of querying the cluster from the command line is as follows:
$curl -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>/<OPERATION_NAME>?<QUERY_STRING>' -d '<BODY>'
Let's understand various parts of this command:
VERB: This can take values for the request method type:GET,POST,PUT,DELETE,HEAD.PROTOCOL: This is eitherhttporhttps.HOST: This is the hostname of the node in the cluster. For local installations...
The queries that we saw until now were basic commands that were used to retrieve data, but the actual power of Elasticsearch's querying lies in a robust Query Domain Specific Language based on JSON also called Query DSL. Kibana makes extensive use of Query DSL in order to get results in a desired format for you. You almost never really have to worry about writing the query JSON, as Kibana will automatically create and put the results in a nice format.
For example, in order to get only three results out of all the matching ones, we can specify it like this:
curl -XPOST 'localhost:9200/logstash-*/_search' -d ' { "query": { "match_all": {} }, "size": 3 }'
The response is as follows, which contains three documents matching the search:
{
"took" : 390,
"timed_out" : false,
"_shards" : {
"total" : 640,
"successful" : 640,
"failed" : 0
},
"hits" : {
"total" : 128,
"max_score" : 1.0,
"hits" : [{
"_index" : "logstash-2014.07.01",...Elasticsearch has a very rich set of plugins, mainly community driven, which are really helpful to analyze the cluster, and execute full-text structural queries easily.
Let's look at a few of the plugins.
This plugin was developed by Lukas Vlcek. It helps analyze the nodes across the cluster with the help of live charts and various statistics related to JVM, CPU, and OS, and about shards and their replicas.
Note
More information is available at https://github.com/lukas-vlcek/bigdesk.
The following screenshot shows the Bigdesk plugin:
BigDesk plugin
The Elastic-Hammer plugin acts as a frontend for Elasticsearch. It helps query the cluster and provides syntax checking while typing queries as well.
Note
More details can be found here: https://github.com/andrewvc/elastic-hammer.
Elasticsearch Elastic-Hammer plugin
In this chapter, we learned the basic concepts of Elasticsearch. We also figured out how querying on a Elasticsearch index works, and how Kibana makes use of Elasticsearch queries to efficiently analyze indexed data and show beautiful visualizations on top of it.
In the next chapter, we will look at Kibana's features in more detail to understand how it helps perform some searches on data with querying on its Discover page.
The rest of the chapter is locked
You have been reading a chapter from
Learning ELK StackPublished in: Nov 2015Publisher: ISBN-13: 9781785887154
© 2015 Packt Publishing Limited All Rights ReservedRegister for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
undefined
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $15.99/month. Cancel anytime
Author (1)
