In this chapter, we will cover the following topics:
Using the enumeration type
Removing HTML tags during indexing
Storing data outside of Solr index
Using synonyms
Stemming different languages
Using nonaggressive stemmers
Using the n-gram approach to do performant trailing wildcard searches
Using position increment to divide sentences
Using patterns to replace tokens
The process of data indexing can be divided into parts. One of the parts is data analysis. It's one of the crucial parts of data preparation. It defines how your data will be divided into terms from text, and what type it will be. The Solr data parsing behavior is defined by types. A type's behavior can be defined in the context of the indexing process, query process, or both. Furthermore, the type definition is composed of a tokenizer (or multiple tokenizers, some for querying and some for indexing) and filters (both token and character filters). A tokenizer specifies how your data will be preprocessed after it is sent to the appropriate field. An analyzer operates on the whole data that is sent to the field. Types can only have one tokenizer. The result of the tokenizer is a stream of objects called tokens.
Next in the analysis chain are the filters. They operate on the tokens in the token stream. They can do anything with the tokens—changing, removing, or making them lowercase...
Imagine that we use Solr to store information about our environment's state, error, and events related to them—a simple solution that will work as a simple log centralization solution. For our simple use case, we will store the identifier of the message, the information, what type of event it is, and the severity of the event, showing us how important the event is. However, what we will want to be sure of is that the severity field contains only values from a given list. To achieve all this, we will use the Solr enumeration type.
To achieve our requirements, we will have to perform the following steps:
We will start with the index structure. Our field list from the schema.xml file will look as follows:
<field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="problem" type="text_general" indexed="true" stored="true" /> <field name="severity" type="enum_type" indexed="true" stored="true" />
In addition...
Although Solr allows us to use the partial update API to update a single field of our document, what it does in the background is the complete reindexing of a document. However, there are situations where such reindexing is not possible. For example, we can have an index containing articles about published books, and we can store the information on how many users visited this article and read it. The number of users is so high that we have thousands of updates per second. Sending a high amount of updates can be demanding for Solr; however, we can store such information in external files and use it for boosting or sorting. This recipe will show how to do this.
Let's assume we have an e-commerce client and we are providing a search system based on Solr. Our index has thousands of documents that mainly consist of books and everything works fine. Then, one day, someone from the marketing department comes into your office and says that he wants to be able to find books that have the word machine when he types electronics into the search box. The first thing that comes to mind is "hey, do it in the source and I'll index that". However, this is not an option this time because there can be many documents in the database that have those words. We don't want to change the whole database. This is when synonyms come into play, and this recipe will show you how to use synonyms.
Stemming is a very common requirement; it is the process of reducing words to their root form (or stems). Let's imagine the book e-commerce store, where you store the books' names and descriptions. We want to be able to find words such as shown and showed when you type the word show, and vice versa. We can achieve this requirement using stemming algorithms. The thing is, there are no general stemmers; they are language-specific. This recipe will show you how to add stemming to your data analysis chain and where to look for a list of stemmers.
To achieve our requirement to stem English, we need to take certain steps:
We will start with the index structure. Let's assume that our index consists of three fields that we defined in the schema.xml file:
<field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="name" type="string" indexed="true" stored="true" /> <field name="description" type="text_stem" indexed...
Nowadays, it's nice to have stemming algorithms (algorithms that will reduce words to their stem or root forms) in your application, which will allow you to find words such as cat and cats just by typing cat. However, let's imagine that you have a search engine that searches through contents of the books in a library. One of the requirements is changing the plural forms of the words from plural to singular; nothing less, nothing more. Can Solr do this? Yes, Solr can do this, and this recipe will show you how to do it.
First, let's start with a simple, two-field index (add the following section to your schema.xml file):
<field name="id" type="string" indexed="true" stored="true" required="true" /> <field name="description" type="text_light_stem" indexed="true" stored="true" />
Now, let's define the text_light_stem field type, which should look like this (add this to your schema.xml file):
<fieldType name="text_light_stem" class="solr...Many users working with traditional RDBMS systems are used to wildcard searches. The most common among them are the ones using the * characters, which means zero or more characters. If you used SQL databases, you probably saw searches such as this:
AND name LIKE 'ABC12%'
However, wildcard searchers are not too efficient when it comes to Solr. This is because Solr needs to enumerate all the terms because the query is executed. So, how do we prepare our Solr deployment to handle trailing wildcard characters in an efficient way? This recipe will show you how to prepare your data and make efficient searches.
There are some steps we need to make efficient wildcards using the n-gram approach:
The first step is to create a proper index structure. Let's assume we have the following fields defined in the schema.xml file:
<field name="id" type="string" indexed="true" stored="true" required="true" /> <field name...
Imagine that we want to search in the short notes created by our users. We want to have two possibilities—searching inside a single sentence and searching inside the whole content of the note. We also know that our users don't write notes longer than 100 sentences, and each sentence has a maximum of 100 words, giving us a maximum of 10,000 words per note. To achieve this, we will use position increments that allow us to control how data is divided in the same field.
The following steps will allow us to fulfill our requirements:
We start with example data, which will look like this:
<add> <doc> <field name="id">1</field> <field name="note_line">Support meeting at Monday.</field> <field name="note_line">Need to prepare presentation.</field> </doc> </add>
Now, we need to create an index structure. To do this, we need to add the fields that will be used. We do this by adding...
Let's assume that we want to search inside user blog posts. We need to prepare a simple search returning only the identifier of the documents that were matched. However, we will want to remove some words because of explicit language. Of course, we can do this using the stop words functionality, but what if we want to know how many documents have their contents censored with compute statistics on. In such a case, we can't use the stop words functionality, we need something more, which means that we need regular expressions. This recipe will show you how to achieve such requirements using Solr and one of its filters.
To achieve our needs, we will use the solr.PatternReplaceFilterFactory filter. Let's assume that we want to remove all the words that start with the word prefix. These are the steps needed:
First, we need to create our index structure, so the fields we add to the schema.xml file are as follows:
<field name="id" type="string" indexed...
© 2015 Packt Publishing Limited All Rights Reserved
