With the growth of data for searching, it becomes necessary to scale up the performance of search applications, to cater to the increasing needs of indexing and searching quickly over large datasets. Distributed search can be used when a single index store becomes difficult to operate in terms of its size (large to fit in memory or disk). As more number of users start using enterprise search, single node searches have limitations in terms of response time and parallel sessions for users. For smaller data sizes, standalone search architecture performs better compared to distributed searches, due to single index availability. However, with the growth in the data size, its performance degrades eventually. The Distributed Search application increases the operation and maintenance cost. It also increases the complexity of overall landscape. However, with the scaling of information for searching, distributed search is the way to forward...
The decision to move to a distributed search from a standalone system should be driven by the needs of enterprises, because distributed search applications are not always efficient in terms of performance. In this section, we will focus on understanding distributed search patterns and how Apache Solr supports distributed search. There have been efforts made to enable Apache Solr work with Apache Hadoop platform in the past, and we will look at more details in coming chapters.
There are two important functions of any enterprise search: creation of indexes and run-time searching on indexes. Any or either of these functions can run in distributed mode, depending upon the requirements from an enterprise. To utilize the distributed search, the indexing must be split into multiple shards and should be kept across multiple nodes of a distributed system.
SolrCloud provides a new way to enable distributed enterprise search using a Apache Solr in enterprises. Previously, with the standard distributed Solr support, lot of the manual work had been automated by SolrCloud. With the introduction of SolrCloud, the manual steps like configuring solr-config.xml to talk with shards, adding documents to the shards, and so on, work became automatic. Unlike the traditional approach of master-slave based distributed Solr, SolrCloud provides a leader-replica-based approach as its implementation. SolrCloud runs on top of Apache ZooKeeper. First, let's understand the ZooKeeper.
SolrCloud contains a cluster of nodes, which use Apache ZooKeeper to talk with each other. Apache ZooKeeper is responsible for maintaining co-ordination among various nodes. Besides co-ordinating among nodes, it also maintains configuration information, and group services to the distributed system. Due to its in-memory management of information;...
In an enterprise, data is generated from all the software that is participating in day-to-day operations. This data has different formats, and bringing in this data for big-data processing requires a storage system that is flexible enough to accommodate a data with varying data models. A NoSQL database, by its design, is best suited for this kind of storage requirements. One of the primary objectives of NoSQL is horizontal scaling, that is, the P in CAP theorem, but this works at the cost of sacrificing Consistency or Availability. Visit http://en.wikipedia.org/wiki/CAP_theorem to understand more about CAP theorem.
In this chapter, we have understood the distributed aspects of any enterprise search. We understood distributed search patterns, and how Apache Solr can be used as a distributed search. We started working with Apache SolrCloud, by understanding its architecture, and building a SolrCloud instance of development and production. We also looked at sharding strategies and fault tolerance. Finally, we went through Apache Solr and MongoDB together. In the coming chapter, we will see how Apache Hadoop and Solr can complement each other, alongside the various implementations of Solr with Hadoop.
© 2015 Packt Publishing Limited All Rights ReservedAuthor (1)
