Apache Solr Enterprise Search Server - Third Edition

Before describing Solr, it is best to start with Apache Lucene, the core technology underlying it. Lucene is an open source, high-performance text search engine library. Lucene was developed and open sourced by Doug Cutting in 2000 and has evolved and matured since then with a strong online community. It is the most widely deployed search technology today. Being just a code library, Lucene is not a server and certainly isn't a web crawler either. This is an important fact. There aren't even any configuration files.

In order to use Lucene, you write your own search code using its API, starting with indexing documents that you supply to it. A document in Lucene is merely a collection of fields, which are name-value pairs containing text or numbers. You configure Lucene with a text analyzer that will tokenize a field's text from a single string into a series of tokens (words) and further transform them by reducing them to their stems, called stemming, substitute synonyms, and/or perform other processing. The final indexed tokens are said to be the terms. The aforementioned process starting with the analyzer is referred to as text analysis. Lucene indexes each document into its index stored on a disk. The index is an inverted index, which means it stores a mapping of a field's terms to associated documents, along with the ordinal word position from the original text. Finally, you search for documents with a user-provided query string that Lucene parses according to its syntax. Lucene assigns a numeric relevancy score to each matching document and only the top scoring documents are returned.

Lucene's major features are:

Apache Solr is an enterprise search server that is based on Lucene. Lucene is such a big part of what defines Solr that you'll see many references to Lucene directly throughout this book. Developing a high-performance, feature-rich application that uses Lucene directly is difficult and it's limited to Java applications. Solr solves this by exposing the wealth of power in Lucene via configuration files and HTTP parameters, while adding some features of its own. Some of Solr's most notable features beyond Lucene are as follows:

Also, there are two contrib modules that ship with Solr that really stand out, which are as follows:

As of the 3.1 release, there is a tight relationship between the Solr and Lucene projects. The source code repository, committers, and developer mailing list are the same, and they are released together using the same version number. Since Solr is always based on the latest version of Lucene, most improvements in Lucene are available in Solr immediately.

There's a good chance that you are unfamiliar with Lucene or Solr and you might be wondering what the fundamental differences are between it and a database. You might also wonder if you use Solr, do you need a database.

The most important comparison to make is with respect to the data model—the organizational structure of the data. The most popular category of databases is relational databases—RDBMS. A defining characteristic of relational databases is a data model, based on multiple tables with lookup keys between them and a join capability for querying across them. That approach has proven to be versatile, being able to satisfy nearly any information-retrieval task in one query.

However, it is hard and expensive to scale them to meet the requirements of a typical search application consisting of many millions of documents and low-latency response. Instead, Lucene has a much more limiting document-oriented data model, which is analogous to a single table. Document-oriented databases such as MongoDB are similar in this respect, but their documents can be nested, similar to XML or JSON. Lucene's document structure is flat like a table, but it does support multivalued fields—a field with an array of values. It can also be very sparse such that the actual fields used from one document to the next vary; there is no space or penalty for a document to not use a field.

Taking a look at the Solr feature list naturally reveals plenty of search-oriented technology that databases generally either don't have, or don't do well. The notable features are relevancy score ordering, result highlighting, query spellcheck, and query-completion. These features are what drew you to Solr, no doubt. And let's not forget faceting. This is possible with a database, but it's hard to figure out how, and it's difficult to scale. Solr, on the other hand, makes it incredibly easy, and it does scale.

Can Solr be a substitute for your database? You can add data to it and get it back out efficiently with indexes; so on the surface, it seems plausible. The answer is that you are almost always better off using Solr in addition to a database. Databases, particularly RDBMSes, generally excel at ACID transactions, insert/update efficiency, in-place schema changes, multiuser access control, bulk data retrieval, and they have second-to-none integration with application software stacks and reporting tools. And let's not forget that they have a versatile data model. Solr falls short in these areas.

When you unzip Solr after downloading it, you should find a relatively straightforward directory structure (differences between Solr 4 and 5 are briefly explained here):

Solr ships with a number of example collection configurations. We're going to run one called techproducts. This example will create a collection and insert some sample data.

First, go to the bin directory, and then run the main Solr command. On Windows, it will be solr.cmd, on *nix systems it will be just solr. Jetty's start.jar file by typing the following command:

>>cd bin
>>./solr start –e techproducts

The >> notation is the command prompt and is not part of the command. You'll see a few lines of output as Solr is started, and then the techproducts collection is created via an API call. Then the sample data is loaded into Solr. When it's done, you'll be directed to the Solr admin at http://localhost:8983/solr.

To stop Solr, use the same Solr command script:

>>./solr stop

Solr comes with some sample data found at example/exampledocs. We saw this data loaded as part of creating the techproducts Solr core when we started Solr. We're going to use that for the remainder of this chapter so that we can explore Solr more, without getting into schema design and deeper data loading options. For the rest of the book, we'll base the examples on the digital supplement to the book—more on that later.

We're going to re-index the example data by using the post.jar Java program, officially called SimplePostTool. Most JAR files aren't executable, but this one is. This simple program takes a Java system variable to specify the collection: -Dc=techproducts, iterates over a list of Solr-formatted XML input files, and HTTP posts it to Solr running on the current machine —http://localhost:8983/solr/techproducts/update. Finally, it will send a commit command, which will cause documents that were posted prior to the commit to be saved and made visible. Obviously, Solr must be running for this to work. Here is the command and its output:

>> cd example/exampledocs
>> java –Dc=techproducts -jar post.jar *.xml
SimplePostTool version 5.0.0
Posting files to [base] url http://localhost:8983/solr/techproducts/update using content-type application/xml...
POSTing file gb18030-example.xml
POSTing file hd.xml
etc.
14 files indexed.
COMMITting Solr index changes to http://localhost:8983/solr/techproducts/update...

If you are using a Unix-like environment, you have an alternate option of using the /bin/post shell script, which wraps the SimplePostTool.

Let's take a look at one of these XML files we just posted to Solr, monitor.xml:

<add>
  <doc>
    <field name="id">3007WFP</field>
    <field name="name">Dell Widescreen UltraSharp 3007WFP</field>
    <field name="manu">Dell, Inc.</field>
    <!-- Join -->
    <field name="manu_id_s">dell</field>
    <field name="cat">electronics</field>
    <field name="cat">monitor</field>
    <field name="features">30" TFT active matrix LCD, 2560 x 1600, .25mm dot pitch, 700:1 contrast</field>
    <field name="includes">USB cable</field>
    <field name="weight">401.6</field>
    <field name="price">2199</field>
    <field name="popularity">6</field>
    <field name="inStock">true</field>
    <!-- Buffalo store -->
    <field name="store">43.17614,-90.57341</field>
  </doc>
</add>

The XML schema for files that can be posted to Solr is very simple. This file doesn't demonstrate all of the elements and attributes, but it shows the essentials. Multiple documents, represented by the <doc> tag, can be present in series within the <add> tag, which is recommended for bulk data loading scenarios. This subset may very well be all that you use. More about these options and other data loading choices will be discussed in Chapter 4, Indexing Data.

Point your browser to http://localhost:8983/solr/#/techproducts/query—this is the query form described in the previous section. The search box is labeled q. This form is a standard HTML form, albeit enhanced by JavaScript. When the form is submitted, the form inputs become URL parameters to an HTTP GET request to Solr. That URL and Solr's search response is displayed to the right. It is convenient to use the form as a starting point for developing a search, but then subsequently refine the URL directly in the browser instead of returning to the form.

Run a query by replacing the *:* in the q field with the word lcd, then clicking on the Execute Query button. At the top of the main content area, you will see a URL like this http://localhost:8983/solr/techproducts/select?q=monitor&wt=json&indent=true. The URL specifies that you want to query for the word lcd, and that the output should be in indented JSON format.

Below this URL, you will see the search result; this result is the response of that URL.

By default, Solr responds in XML, however the query interface specifies JSON by default. Most modern browsers, such as Firefox, provide a good JSON view with syntax coloring and hierarchical controls. All response formats have the same basic structure as the JSON you're about to see. More information on these formats can be found in Chapter 4, Indexing Data.

The JSON response consists of a two main elements: responseHeader and response. Here is what the header element looks like:

"responseHeader": {
    "status": 0,
    "QTime": 1,
    "params": {
      "q": "lcd",
      "indent": "true",
      "wt": "json"
    }
  }
…

The following are the elements from the preceding code snippet:

Next up is the most important part, the results:

"response": {
    "numFound": 5,
    "start": 0,

The numFound value is the number of documents matching the query in the entire index. The start parameter is the index offset into those matching (ordered) documents that are returned in the response below.

Often, you'll want to see the score of each matching document. The document score is a number that represents how relevant the document is to the search query. This search response doesn't refer to scores because it needs to be explicitly requested in the fl parameter—a comma-separated field list. A search that requests the score via fl=*,score will have a maxScore attribute in the "response" element, which is the maximum score of all documents that matched the search. It's independent of the sort order or result paging parameters.

The content of the result element is a list of documents that matched the query. The default sort is by descending score. Later, we'll do some sorting by specified fields.

{
        "id": "9885A004",
        "name": "Canon PowerShot SD500",
        "manu": "Canon Inc.",
        "manu_id_s": "canon",
        "cat": [
          "electronics",
          "camera"
        ],
        "features": [
          "3x zoop, 7.1 megapixel Digital ELPH",
          "movie clips up to 640x480 @30 fps",
          "2.0\" TFT LCD, 118,000 pixels",
          "built in flash, red-eye reduction"
        ],
        "includes": "32MB SD card, USB cable, AV cable, battery",
        "weight": 6.4,
        "price": 329.95,
        "price_c": "329.95,USD",
        "popularity": 7,
        "inStock": true,
        "manufacturedate_dt": "2006-02-13T15:26:37Z",
        "store": "45.19614,-93.90341",
        "_version_": 1500358264225792000
      },
...

The document list is pretty straightforward. By default, Solr will list all of the stored fields. Not all of the fields are necessarily stored—that is, you can query on them but not retrieve their value—an optimization choice. Notice that it uses the basic data types of strings, integers, floats, and Booleans. Also note that certain fields, such as features and cat are multivalued, as indicated by the use of [] to denote an array in JSON.

This was a basic keyword search. As you start using more search features such as faceting and highlighting, you will see additional information following the response element.

Let's take a look at the statistics available via the Plugins / Stats page. This page provides details on all the components of Solr. Browse to CORE and then pick a Searcher. Before we loaded data into Solr, this page reported that numDocs was 0, but now it should be 32.

Now take a look at the update handler stats by clicking on the UPDATEHANDLER and then expand the stats for the update handler by clicking on the updateHandler toggle link on the right-hand side of the screen. Notice that the /update request handler has some stats too:

If you think of Solr as a RESTful server, then the various public end points are exposed under the QUERYHANDLER menu. Solr isn't exactly REST-based, but it is very similar. Look at the /update to see the indexing performance, and /select for query performance.

The final destination of our quick Solr tour is to visit the so-called browse interface—available at http://localhost:8983/solr/techproducts/browse. It's for demonstrating various Solr features:

This is also a demonstration of Solritas, which formats Solr requests using templates that are based on Apache Velocity. The templates are VM files in example/techproducts/solr/techproducts/conf/velocity. Solritas is primarily for search UI prototyping. It is not recommended for building anything substantial. See Chapter 9, Integrating Solr, for more information.

Here is a screenshot of the browse interface; not all of it is captured in this image:

In what ways do you need your data to be searched? Will you need faceted navigation, spelling suggestions, or more-like-this capabilities? Knowing your requirements up front is the key in producing a well-designed search solution. Understanding how to implement these features is critical. A well-designed schema lays the foundation for a successful Solr implementation.

However, during the development cycle, having the flexibility to try different field types without changing the schema and restarting Solr can be very handy. The dynamic fields feature allows you to assign field types by using field name conventions during indexing. Solr provides many useful predefined dynamic fields. Chapter 2, Schema Design, will cover this in-depth.

However, you can also get started right now. Take a look at the stock dynamic fields in /server/solr/configsets/sample_techproducts_configs/conf/schema.xml. The dynamicField, XML tags represent what is available. For example, the dynamicField named *_b allows you to store and index Boolean data types; a field named admin_b would match this field type.

For the stock dynamic fields, here is a subset of what's available from the schema.xml file:

To make use of these fields, you simply name your fields using those suffixes—example/exampledocs/ipod_other.xml makes good use of the *_dt type with its manufacturedate_dt field. Copying an example file, adding your own data, changing the suffixes, and indexing (via the SimplePost tool) is all as simple as it sounds. Give it a try!

It's probably a good time to talk a little more about text analysis. When considering field types, it's important to understand how your data is processed. For each field, you'll need to know its data type, and whether or not the value should be stored and/or indexed. For string types, you'll also need to think about how the text is analyzed.

Simply put, text analysis is the process of extracting useful information from a text field. This process normally includes two steps: tokenization and filtering. Analyzers encapsulate this entire process, and Solr provides a way to mix and match analyzer behaviors by configuration.

Tokenizers split up text into smaller chunks called tokens. There are many different kinds of tokenizers in Solr, the most common of which splits text on word boundaries, or whitespace. Others split on regular expressions, or even word prefixes. The tokenizer produces a stream of tokens, which can be fed to an optional series of filters.

Filters, as you may have guessed, commonly remove noise—things such as punctuation and duplicate words. Filters can even lower/upper case tokens, and inject word synonyms.

Once the tokens pass through the analyzer processor chain, they are added to the Lucene index. Chapter 2, Schema Design, covers this process in detail.

The next step is, naturally, searching. For most applications processing user queries, you will want to use the [e]dismax query parser, set with defType=edismax. It is not the default but arguably should be in our opinion; [e]dismax handles end-user queries very well. There are a few more configuration parameters it needs, described in Chapter 5, Searching.

Here are a few example queries to get you thinking.

Find all the documents that have the phrase hard drive in their cat field:

http://localhost:8983/solr/techproducts/select?q=cat:"hard+drive"

Find all the documents that are in-stock, and have a popularity greater than 6:

http://localhost:8983/solr/techproducts/select?q=+inStock:true+AND+popularity:[6+TO+*]

Here's an example using the eDisMax query parser:

http://localhost:8983/solr/techproducts/select?q=ipod&defType=edismax&qf=name^3+manu+cat&fl=*,score

This returns documents where the user query in q matches the name, manu, and cat fields. The ^3 after the manu field tells Solr to boost the relevancy of the document scores when the manu field matches. The fl param tells Solr what fields to return—The * means return all fields, and score is a number that represents how well the document matched the input query.

Faceting and statistics can be seen in this example:

http://localhost:8983/solr/techproducts/select?q=ipod&defType=dismax&qf=name^3+manu+cat&fl=*,score&rows=0&facet=true&facet.field=manu_id_s&facet.field=cat&stats=true&stats.field=price&stats.field=weight

This builds on the previous, dismax example, but instead of returning documents (rows=0), Solr returns multiple facets and stats field values.

For detailed information on searching, see Chapter 5, Searching.

If the previous tips on indexing and searching are enough to get you started, then you must be wondering how you integrate Solr and your application. By far, the most common approach is to communicate with Solr via HTTP. You can make use of one of the many HTTP client libraries available. Here's a small example using the Ruby library, RSolr:

require "rsolr"
client = RSolr.connect
params = {:q => "ipod", :defType => "dismax", :qf => "name^3 manu cat", :fl => "*,score"}
result = client.select(:params => params)
result["response"]["docs"].each do |doc|
  puts doc.inspect
end

Using one of the previous sample queries, the result of this script would print out each document, matching the query ipod.

There are many client implementations, and finding the right one for you is dependent on the programming language your application is written in. Chapter 9, Integrating Solr, covers this in depth, and will surely set you in the right direction.