Chapter 7. Apache Nutch

In the previous chapter, we saw how we can index documents using Apache Tika into Solr. In this chapter, we'll see how we can use Apache Nutch to index web content into Solr and index them in Solr. This chapter will cover the following topics:

• Introducing to Apache Nutch

• Installing Apache Nutch

• Configuring Solr with Nutch

Introducing Apache Nutch

Apache Nutch is an open source web crawler that can be used to retrieve data from websites and get data from it. It is an extensible and scalable crawler that gives us the freedom to use it as we like by using plugins. Apache Nutch is written in Java, just like Apache Solr, and both tools make a perfect combination for creating a search engine of our own if they are combined.

Apache Nutch can be used on a single node or can be run in a distributed way with multiple nodes. Let's see how we can combine Apache Solr and Apache Nutch to crawl a web page and index it. To do this, let's start by installing Apache Nutch.

Installing Apache Nutch

Apache Nutch comes in two versions (1.x and 2.x). For this example, we'll be using version 1.x, as it contains a binary that will help reduce the time taken to build version 2.x from scratch. The latest stable version of Apache Nutch (v1.10), which also contains a binary at the time of writing this book, can be installed by following these steps:

1. Download and unzip Apache Nutch (apache-nutch-1.10-bin.tar.gz) from http://nutch.apache.org/downloads.html.

2. Extract the archive file into a folder of your choice. We'll use %NUTCH_HOME% as the folder where the ZIP file is to be extracted.

   

Note

On Windows, we can install Cygwin by going to the installation link at http://cygwin.com/install.html.

Let's verify the downloaded archive by going to %NUTCH_HOME%/bin. It will contain the Nutch script, which we can execute. We run the following command to get a list of available options that we can use:

$ cd %NUTCH_HOME%/bin
$ ./nutch

We should get the following output from the command:

Usage...

Configuring Solr with Nutch

Apache Solr can easily be configured for use with Nutch. We can perform the following steps to integrate Apache Nutch with Solr:

1. Create a new core (nutch-example) in Solr by copying the nutch-example folder from the Chapter 7 code that comes with this book.

2. After creating the new core, we just need to restart the Solr instance.

3. After we have restarted the Solr instance, let's crawl some data using Nutch and index it into Solr. To do this, we'll navigate to the %NUTCH_HOME% folder and execute the following command:

   $ bin/crawl
   

   After executing the command, we'll see the following output:

   Usage: crawl [-i|--index] [-D "key=value"] <Seed Dir> <Crawl Dir> <Num Rounds>
           -i|--index      Indexes crawl results into a configured indexer
           -D              A Java property to pass to Nutch calls
           Seed Dir        Directory in which to look for a seeds file
           Crawl Dir       Directory where the crawl/link/segments dirs are saved
         ...

Summary

In this chapter, we saw how we can use Apache Nutch with Solr and combine them to create a simple search engine that will crawl the website. We also indexed the crawl data that we obtained from Nutch into Solr using the commands provided by Apache Nutch.

In the next chapter, we'll see how we can use index data in real time using the inbuilt features of Solr.