Chapter 9. Advanced Topics – Multilanguage, Deduplication, and Others

In the previous chapter, we saw how we can use Solr to retrieve documents that are indexed into it in real time. In this chapter, we'll cover some advanced topics that will help us use the full potential of Solr.

Specifically, we'll cover the following topics in this chapter:

• Indexing a document in multiple languages

• Detecting duplicate documents (deduplication)

• Streaming of documents in Solr (content streaming)

• UIMA integration with Solr

Multilanguage indexing

Solr provides us with a way to index multilanguage documents in it. In this section, we'll cover how to easily index multilanguage documents in Solr and also how to auto-detect a document language.

Let's create a new core called languages-example. It will contain the following fields in schema.xml, which we're going to use for our example:

<fields>
  <field name="id" type="string" indexed="true" stored="true" required="true"/>
  <field name="content" type="text_general" indexed="true" stored="true" />
  <field name="text" type="text_general" multiValued="true" indexed="true" stored="false" />
  <copyField source="content" dest="text" />
  <field name="language" type="string" stored="true" indexed="true" />

  <dynamicField name="*_en" type="text_en" stored="true" indexed="true" />
  <dynamicField name="*_ru" type="text_ru" stored="true" indexed="true" />
  <dynamicField name="*_fr" type="text_fr" stored="true" indexed...

Removing duplicate documents (deduplication)

Solr provides us with a way to prevent duplicate or nearly duplicate elements to get indexed using a signature/fingerprint field. It natively provides a deduplication technique of this type via the signature class, and this can further be used to implement new hash and signature implementations.

Let's see how we can implement deduplication in Solr. We'll use our musicCatalog core, which we used in the previous chapter as well, and will modify it:

1. Copy the musicCatalog core and create a new core called musicCatalog-dedupe from it. After we have created the new core, we'll change schema.xml to add a signature field that will contain the document signature/fingerprint:

   <!-- Field to store the fingerprint/signature -->
   <field name="signature" type="string" indexed="true" stored="true" required="true" multiValued="false" />

2. After adding the field, we'll add a new UpdateRequestProcessor element to solrconfig.xml configuration file, which will...

Content streaming

In Solr, we can index remote or local files by enabling remote streaming in solrconfig.xml. Let's see how we can use this feature in Solr, we'll follow the steps given here to enable the remote streaming feature.

Let's use our newly created languages-example core and modify solrconfig.xml. We'll replace the requestDispatcher config in our solrconfig.xml file with the following lines:

  <requestDispatcher handleSelect="false" > 
    <requestParsers enableRemoteStreaming="true" 
      multipartUploadLimitInKB="2048000"
      formdataUploadLimitInKB="2048"
      addHttpRequestToContext="false"/> 
  </requestDispatcher>

The enableRemoteStreaming="true" property will enable the remote streaming feature. This will enable us to index remote or local files. Let's go ahead and index a remote file in our Solr index:

$ curl http://localhost:8983/solr/languages-example/update?commit=true -F stream.url=https://raw.githubusercontent.com/sachin-handiekar/SolrIndexingBook...

UIMA integration with Solr

Solr can also be integrated with Apache UIMA (short for Unstructured Information Management Architecture), which can be used to define a custom pipeline to add metadata to documents.

Note

More information about Solr UIMA integration can be found at https://wiki.apache.org/solr/SolrUIMA.

In Solr, UIMA can be configured by following these steps:

1. In solrconfig.xml, we can add the following libraries:

   <lib dir="../../contrib/uima/lib" />
   <lib dir="../../dist/" regex="solr-uima-\d.*\.jar" />

2. After adding the libraries, we can add the following fields to schema.xml, which will contain the language, concept, and sentence fields:

   <field name="language" type="string" indexed="true" stored="true" required="false"/>
   <field name="concept" type="string" indexed="true" stored="true" multiValued="true" required="false"/>
   <field name="sentence" type="text" indexed="true" stored="true" multiValued="true" required="false" />

3. After adding these fields, we'll...

Summary

In this chapter, we saw how Solr can be used to index multilanguage documents with some easy configuration, and also language detection, which can be used to automatically detect the language of a document. Plus, we covered the deduplication technique, which Solr supports natively. It can be used to overwrite/remove documents from Solr using the hashing/fingerprint technique. We also covered in brief content streaming and the integration of Apache UIMA with Solr.

In the next chapter, we'll cover how we can easily set up a cluster of Solr servers.