To efficiently use ElasticSearch, it is very important to understand how it works.
The goal of this chapter is to give the readers an overview of the basic concepts of ElasticSearch and to be a quick reference for them. It's essential to understand the basics better so that you don't fall into the common pitfall about how ElasticSearch works and how to use it.
The key concepts that we will see in this chapter are: node, index, shard, mapping/type, document, and field.
ElasticSearch can be used both as a search engine as well as a data store.
A brief description of the ElasticSearch logic helps the user to improve performance, search quality, and decide when and how to optimize the infrastructure to improve scalability and availability.
Some details on data replications and base node communication processes are also explained.
At the end of this chapter, the protocols used to manage ElasticSearch are also discussed.
Every instance of ElasticSearch is called a node. Several nodes are grouped in a cluster. This is the base of the cloud nature of ElasticSearch.
To better understand the following sections, some basic knowledge about the concepts of the application node and cluster are required.
One or more ElasticSearch nodes can be set up on a physical or a virtual server depending on the available resources such as RAM, CPU, and disk space.
A default node allows you to store data in it to process requests and responses. (In Chapter 2, Downloading and Setting Up, we'll see details about how to set up different nodes and cluster topologies).
When a node is started, several actions take place during its startup, such as:
The configuration is read from the environment variables and the
elasticsearch.ymlconfiguration fileA node name is set by the configuration file or is chosen from a list of built-in random names
Internally, the ElasticSearch engine initializes all the modules and plugins that are available in the current installation
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
After the node startup, the node searches for other cluster members and checks its index and shard status.
To join two or more nodes in a cluster, the following rules must be observed:
The version of ElasticSearch must be the same (v0.20, v0.9, v1.4, and so on) or the join is rejected.
The cluster name must be the same.
The network must be configured to support broadcast discovery (it is configured to it by default) and they can communicate with each other. (See the Setting up networking recipe in Chapter 2, Downloading and Setting Up.)
A common approach in cluster management is to have a master node, which is the main reference for all cluster-level actions, and the other nodes, called secondary nodes, that replicate the master data and its actions.
To be consistent in the write operations, all the update actions are first committed in the master node and then replicated in the secondary nodes.
In a cluster with multiple nodes, if a master node dies, a master-eligible node is elected to be the new master node. This approach allows automatic failover to be set up in an ElasticSearch cluster.
There are two important behaviors in an ElasticSearch node: the non-data node (or arbiter) and the data container behavior.
Non-data nodes are able to process REST responses and all other operations of search. During every action execution, ElasticSearch generally executes actions using a map/reduce approach: the non-data node is responsible for distributing the actions to the underlying shards (map) and collecting/aggregating the shard results (redux) to be able to send a final response. They may use a huge amount of RAM due to operations such as facets, aggregations, collecting hits and caching (such as scan/scroll queries).
Data nodes are able to store data in them. They contain the indices shards that store the indexed documents as Lucene (internal ElasticSearch engine) indices.
Using the standard configuration, a node is both an arbiter and a data container.
In big cluster architectures, having some nodes as simple arbiters with a lot of RAM, with no data, reduces the resources required by data nodes and improves performance in searches using the local memory cache of arbiters.
The Setting up different node types recipe in Chapter 2, Downloading and Setting Up.
When a node is running, a lot of services are managed by its instance. These services provide additional functionalities to a node and they cover different behaviors such as networking, indexing, analyzing and so on.
ElasticSearch natively provides a large set of functionalities that can be extended with additional plugins.
During a node startup, a lot of required services are automatically started. The most important are:
Cluster services : These manage the cluster state, intra-node communication, and synchronization.
Indexing Service : This manages all indexing operations, initializing all active indices and shards.
Mapping Service: This manages the document types stored in the cluster (we'll discuss mapping in Chapter 3, Managing Mapping).
Network Services: These are services such as HTTP REST services (default on port 9200), internal ES protocol (port 9300) and the Thrift server (port 9500), applicable only if the Thrift plugin is installed.
Plugin Service: This enables us to enhance the basic ElasticSearch functionality in a customizable manner. (It's discussed in Chapter 2, Downloading and Setting Up, for installation and Chapter 12, Plugin Development, for detailed usage.)
River Service: It is a pluggable service running within ElasticSearch cluster, pulling data (or being pushed with data) that is then indexed into the cluster. (We'll see it in Chapter 8, Rivers.)
Language Scripting Services: They allow you to add new language scripting support to ElasticSearch.
If you are going to use ElasticSearch as a search engine or a distributed data store, it's important to understand concepts of how ElasticSearch stores and manages your data.
To work with ElasticSearch data, a user must have basic concepts of data management and JSON data format, which is the lingua franca to work with ElasticSearch data and services.
Our main data container is called index (plural indices) and it can be considered as a database in the traditional SQL world. In an index, the data is grouped into data types called mappings in ElasticSearch. A mapping describes how the records are composed (fields).
Every record that must be stored in ElasticSearch must be a JSON object.
Natively, ElasticSearch is a schema-less data store; when you enter records in it during the insert process it processes the records, splits it into fields, and updates the schema to manage the inserted data.
To manage huge volumes of records, ElasticSearch uses the common approach to split an index into multiple shards so that they can be spread on several nodes. Shard management is transparent to the users; all common record operations are managed automatically in the ElasticSearch application layer.
Every record is stored in only a shard; the sharding algorithm is based on a record ID, so many operations that require loading and changing of records/objects, can be achieved without hitting all the shards, but only the shard (and its replica) that contains your object.
The following schema compares ElasticSearch structure with SQL and MongoDB ones:
|
ElasticSearch |
SQL |
MongoDB |
|---|---|---|
|
Index (Indices) |
Database |
Database |
|
Shard |
Shard |
Shard |
|
Mapping/Type |
Table |
Collection |
|
Field |
Field |
Field |
|
Object (JSON Object) |
Record (Tuples) |
Record (BSON Object) |
To ensure safe operations on index/mapping/objects, ElasticSearch internally has rigid rules about how to execute operations.
In ElasticSearch, the operations are divided into:
Cluster/index operations: All clusters/indices with active write are locked; first they are applied to the master node and then to the secondary one. The read operations are typically broadcasted to all the nodes.
Document operations: All write actions are locked only for the single hit shard. The read operations are balanced on all the shard replicas.
When a record is saved in ElasticSearch, the destination shard is chosen based on:
The
id(unique identifier) of the record; if theidis missing, it is autogenerated by ElasticSearchIf
routingorparent(we'll see it in the parent/child mapping) parameters are defined, the correct shard is chosen by the hash of these parameters
Splitting an index in shard allows you to store your data in different nodes, because ElasticSearch tries to balance the shard distribution on all the available nodes.
Every shard can contain up to 2^32 records (about 4.9 billion), so the real limit to a shard size is its storage size.
Shards contain your data and during search process all the shards are used to calculate and retrieve results. So ElasticSearch performance in big data scales horizontally with the number of shards.
All native records operations (such as index, search, update, and delete) are managed in shards.
Shard management is completely transparent to the user. Only an advanced user tends to change the default shard routing and management to cover their custom scenarios. A common custom scenario is the requirement to put customer data in the same shard to speed up his operations (search/index/analytics).
It's best practice not to have a shard too big in size (over 10 GB) to avoid poor performance in indexing due to continuous merging and resizing of index segments.
It is also not good to over-allocate the number of shards to avoid poor search performance due to native distributed search (it works as map and reduce). Having a huge number of empty shards in an index will consume memory and increase the search times due to an overhead on network and results aggregation phases.
Shard on Wikipedia http://en.wikipedia.org/wiki/Shard_(database_architecture)
Related to shard management, there is the key concept of replication and cluster status.
You need one or more nodes running to have a cluster. To test an effective cluster, you need at least two nodes (that can be on the same machine).
An index can have one or more replicas; the shards are called primary if they are part of the primary replica, and secondary ones if they are part of replicas.
To maintain consistency in write operations, the following workflow is executed:
The write operation is first executed in the primary shard
If the primary write is successfully done, it is propagated simultaneously in all the secondary shards
If a primary shard becomes unavailable, a secondary one is elected as primary (if available) and then the flow is re-executed
During search operations, if there are some replicas, a valid set of shards is chosen randomly between primary and secondary to improve its performance. ElasticSearch has several allocation algorithms to better distribute shards on nodes. For reliability, replicas are allocated in a way that if a single node becomes unavailable, there is always at least one replica of each shard that is still available on the remaining nodes.
The following figure shows some examples of possible shards and replica configuration:
The replica has a cost in increasing the indexing time due to data node synchronization, which is the time spent to propagate the message to the slaves (mainly in an asynchronous way).
Related to the concept of replication, there is the cluster status indicator that will show you information on the health of your cluster. It can cover three different states:
Green: This shows that everything is okay
Yellow: This means that some shards are missing but you can work on your cluster
Red: This indicates a problem as some primary shards are missing
Mainly, yellow status is due to some shards that are not allocated.
If your cluster is in the recovery status (meaning that it's starting up and checking the shards before they are online), you need to wait until the shards' startup process ends.
After having finished the recovery, if your cluster is always in the yellow state, you may not have enough nodes to contain your replicas (for example, maybe the number of replicas is bigger than the number of your nodes). To prevent this, you can reduce the number of your replicas or add the required number of nodes. A good practice is to observe that the total number of nodes must not be lower than the maximum number of replicas present.
This means you are experiencing lost data, the cause of which is that one or more shards are missing.
To fix this, you need to try to restore the node(s) that are missing. If your node restarts and the system goes back to the yellow or green status, then you are safe. Otherwise, you have obviously lost data and your cluster is not usable; the next action would be to delete the index/indices and restore them from backups or snapshots (if you have done them) or from other sources. To prevent data loss, I suggest having always a least two nodes and a replica set to 1 as good practice.
You can communicate with several protocols using your ElasticSearch server. In this recipe, we will take a look at the main protocols.
ElasticSearch is designed to be used as a RESTful server, so the main protocol is the HTTP, usually on port number 9200 and above. Thus, it allows using different protocols such as native and thrift ones.
Many others are available as extension plugins, but they are seldom used, such as memcached, couchbase, and websocket. (If you need to find more on the transport layer, simply type in Elasticsearch transport on the GitHub website to search.)
Every protocol has advantages and disadvantages. It's important to choose the correct one depending on the kind of applications you are developing. If you are in doubt, choose the HTTP Protocol layer that is the standard protocol and is easy to use.
Choosing the right protocol depends on several factors, mainly architectural and performance related. This schema factorizes advantages and disadvantages related to them. If you are using any of the protocols to communicate with ElasticSearch official clients, switching from a protocol to another is generally a simple setting in the client initialization.
|
Protocol |
Advantages |
Disadvantages |
Type |
|---|---|---|---|
|
HTTP |
|
|
|
|
Native |
|
|
|
|
Thrift |
|
|
This recipe shows us the usage of the HTTP protocol with an example.
You need a working instance of the ElasticSearch cluster. Using default configuration, ElasticSearch enables port number 9200 on your server to communicate in HTTP.
The standard RESTful protocol is easy to integrate.
We will see how easy it is to fetch the ElasticSearch greeting API on a running server on port 9200 using different programming languages:
In BASH, the request will be:
curl –XGET http://127.0.0.1:9200In Python, the request will be:
import urllib result = urllib.open("http://127.0.0.1:9200")In Java, the request will be:
import java.io.BufferedReader; import java.io.InputStream; import java.io.InputStreamReader; import java.net.URL; … truncated… try{ // get URL content URL url = new URL("http://127.0.0.1:9200"); URLConnection conn = url.openConnection();// open the stream and put it into BufferedReader BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream())); String inputLine; while ((inputLine = br.readLine()) != null){ System.out.println(inputLine); } br.close(); System.out.println("Done"); } Catch (MalformedURLException e) { e.printStackTrace(); } catch (IOException e){ e.printStackTrace(); }In Scala, the request will be:
scala.io.Source.fromURL("http://127.0.0.1:9200","utf-8").getLines.mkString("\n")
For every language sample, the response will be the same:
{
"ok" : true,
"status" : 200,
"name" : "Payge, Reeva",
"version" : {
"number" : "1.4.0",
"snapshot_build" : false
},
"tagline" : "You Know, for Search"
}Every client creates a connection to the server index / and fetches the answer. The answer is a valid JSON object. You can invoke the ElasticSearch server from any language that you like.
The main advantages of this protocol are:
Portability: This uses Web standards so that it can be integrated in different languages (Erlang, JavaScript, Python, Ruby, and so on) or called via a command-line application such as cURL.
Durability: The REST APIs don't change often. They don't break for minor release changes as native protocol does.
Simple to use: This has JSON-to-JSON interconnectivity.
Good support: This has much more support than other protocols. Every plugin typically supports a REST endpoint on HTTP.
Easy cluster scaling: You can simply put your cluster nodes behind an HTTP load balancer to balance the calls such as HAProxy or NGinx.
In this book, a lot of the examples are done by calling the HTTP API via the command-line cURL program. This approach is very fast and allows you to test functionalities very quickly.
ElasticSearch provides a native protocol, used mainly for low-level communication between nodes, but very useful for fast importing of huge data blocks. This protocol is available only for Java Virtual Machine (JVM) languages and commonly is used in Java, Groovy, and Scala.
You need a working instance of the ElasticSearch cluster; the standard port number for native protocol is 9300.
The following are the steps required to use the native protocol in a Java environment (we'll discuss this in depth in Chapter 10, Java Integration):
Before starting, we must be sure that Maven loads the
Elasticsearch.jarfile by adding the following code to thepom.xmlfile:<dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>1.4.1</version> </dependency>
Depending on the
Elasticsearch.jarfile, creating a Java client is quite easy:import org.elasticsearch.common.settings.ImmutableSettings; import org.elasticsearch.common.settings.Settings; import org.elasticsearch.client.Client; import org.elasticsearch.client.transport.TransportClient; … Settings settings = ImmutableSettings.settingsBuilder() .put("client.transport.sniff", true).build(); // we define a new settings // using sniff transport allows to autodetect other nodes Client client = new TransportClient(settings) .addTransportAddress(new InetSocketTransportAddress("127.0.0.1", 9300)); // a client is created with the settings
To initialize a native client, a settings object is required, which contains some configuration parameters. The most important ones are:
cluster.name: This is the name of the clusterclient.transport.sniff: This allows you to sniff out the rest of the cluster and add them into its list of machines to use
With the settings object, it's possible to initialize a new client by giving an IP address and port a number (default 9300).
The native protocol is the internal one used in ElasticSearch. It's the fastest protocol that is available to communicate with ElasticSearch.
The native protocol is an optimized binary and works only for JVM languages; to use this protocol, you need to include the elasticsearch.jar in your JVM project. Because it depends on ElasticSearch implementation, it must be the same version of ElasticSearch cluster.
For this reason, every time you update ElasticSearch, you need to update the elasticsearch.jar file on which it depends and if there are internal API changes, you need to update your code.
To use this protocol, you need to study the internals of ElasticSearch, so it's not as easy to use as HTTP and Thrift protocol.
Native protocol is useful for massive data import. But as ElasticSearch is mainly thought as a REST HTTP server to communicate with, it lacks support for everything that is not standard in the ElasticSearch core, such as the plugin's entry points. So using this protocol, you are unable to call entry points made by external plugins.
Note
The native protocol seems the most easy to integrate in a Java/JVM project. However, due to its nature that follows the fast release cycles of ElasticSearch, it changes very often. Also, for minor release upgrades, your code is more likely to be broken. Thus, ElasticSearch developers wisely tries to fix them in the latest releases.
The native protocol is the most used in the Java world and it will be deeply discussed in Chapter 10, Java Integration and Chapter 12, Plugin Development
Further details on ElasticSearch Java API are available on the ElasticSearch website at http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api/current/index.html
Thrift is an interface definition language, initially developed by Facebook, used to define and create services. This protocol is now maintained by Apache Software Foundation.
Its usage is similar to HTTP, but it bypasses the limit of HTTP protocol (latency, handshake and so on) and it's faster.
You need a working instance of ElasticSearch cluster with the thrift plugin installed (https://github.com/elasticsearch/elasticsearch-transport-thrift/); the standard port for the Thrift protocol is 9500.
To use the Thrift protocol in a Java environment, perform the following steps:
We must be sure that Maven loads the thrift library adding to the
pom.xmlfile; the code lines are:<dependency> <groupId>org.apache.thrift</groupId> <artifactId>libthrift</artifactId> <version>0.9.1</version> </dependency>
In Java, creating a client is quite easy using ElasticSearch generated classes:
import org.apache.thrift.protocol.TBinaryProtocol; import org.apache.thrift.protocol.TProtocol; import org.apache.thrift.transport.TSocket; import org.apache.thrift.transport.TTransport; import org.apache.thrift.transport.TTransportException; import org.elasticsearch.thrift.*; TTransport transport = new TSocket("127.0.0.1", 9500); TProtocol protocol = new TBinaryProtocol(transport); Rest.Client client = new Rest.Client(protocol); transport.open();To initialize a connection, first we need to open a socket transport. This is done with the
TSocket(host, port)setting, using the ElasticSearch thrift standard port 9500.Then the socket transport protocol must be encapsulated in a binary protocol, this is done with the
TBinaryProtocol(transport)parameter.Now, a client can be initialized by passing the protocol. The
Rest.Clientutility class and other utility classes are generated byelasticsearch.thrift. It resides in theorg.elasticsearch.thriftnamespace.To have a fully working client, we must open the socket (
transport.open()).At the end of program, we should close the client connection (
transport.close()).
Some drivers, to connect to ElasticSearch, provide an easy-to-use API to interact with Thrift without the boulder that this protocol needs.
For advanced usage, I suggest the use of the Thrift protocol to bypass some problems related to HTTP limits, such as:
The number of simultaneous connections required in HTTP; Thrift transport efficiently uses resources
The network traffic is light weight because it is binary and is very compact
A big advantage of this protocol is that on the server side it wraps the REST entry points so that it can also be used with calls provided by external REST plugins.
For more details on Thrift visit its Wikipedia page at: http://en.wikipedia.org/wiki/Apache_Thrift
For more complete reference on the Thrift ElasticSearch plugin, the official documentation is available at https://github.com/elasticsearch/elasticsearch-transport-thrift/
