Chapter 6. Solr for Spatial Search

In the previous chapter, we discussed in depth the problems faced during the implementation of Solr for search operations on an e-commerce website. We saw solutions to the problems and areas where optimizations may be necessary. We also took a look at semantic search and how it can be implemented in an e-commerce scenario.

In this chapter, we will explore Solr with respect to spatial search. We will look at different indexing techniques and study query types that are specific to spatial data. We will also learn different scenario-based filtering and searching techniques for geospatial data.

The topics that we will cover in this chapter are:

• Features of spatial search

• Lucene 4 spatial module

• Indexing for spatial search

• Search and filtering on spatial index

• Distance sort and relevance boost

• Advanced concepts

  • Quadtrees

  • Geohash

Features of spatial search

With Solr, we can combine location-based data with normal text data in our index. This is termed spatial search or geospatial search.

Earlier versions of Solr (Solr 3.x) provided the following features for spatial search:

• Representation of spatial data as latitude and longitude in Solr

• Filtering by geofilt and bound box filters

• Use of the geodist function to calculate distance

• Distance-based faceting and boosting of results

With Solr 4, the following new features have been introduced in Solr:

• Support for new shapes: Polygon, LineString, and other new shapes are supported as indexed and query shapes in Solr 4. Shapes other than points, rectangles, and circles are supported via the Java Topology Suite (JTS), an optional dependency that we will discuss later.

• Indexing multi-valued fields: This is critical for storing the results of automatic place extraction from text using natural language processing techniques, since a variable number of locations will be found for a...

Lucene 4 spatial module

Solr 4 contains three field types for spatial search: LatLonType (or its non-geodetic twin PointType), SpatialRecursivePrefixTreeFieldType (RPT for short), and BBoxField (to be introduced in Solr 4.10 onward). LatLonType has been there since Lucene 3. RPT offers more features than LatLonType and offers fast filter performance. LatLonType is more appropriate for efficient distance sorting and boosting. With Solr, we can use both the fields simultaneously—LatLonType for sorting or boosting and RPT for filtering. BBoxField is used for indexing bounding boxes, querying by a box, specifying search predicates such as Intersects, Within, Contains, Disjoint, or Equals, and relevancy sorting or boosting of properties such as overlapRatio.

We have already seen the LatLonType field, which we used to define the location of our store in the earlier examples. Let us explore RPT and have a look at BBoxField.

SpatialRecursivePrefixTreeFieldType

RPT available in Solr 4 is used to implement...

Indexing for spatial search

Now that we know the features supported by Solr for spatial search, let us see how indexing should be done for spatial search. We will need to index the coordinates for point, circle, rectangle, and other spatial representations in Solr as documents before we execute a search for them. Every geographical point is represented as latitude and longitude with the format lat,lon.

We saw points being indexed in the last section in our location.csv file. They are indexed in the lat,lon format:

<field name="store">28.570485,77.324713</field>

The points can be indexed in the lat-lon format by omitting the comma:

<field name="store">28.570485 77.324713</field>

We saw that Solr also supports different geometrical shapes such as rectangle, circle, and polygon. Let us learn how to index these.

In order to index a rectangle, we need the two points that form the starting and the ending points of the rectangle. Consider a two-dimensional coordinate system with...

Searching and filtering on a spatial index

Spatial fields in a Solr index can be searched and filtered using the {!geofilt} and {!bbox} query filters. These filters were introduced and are available in Solr 4.2 onward. We saw a working example of geofilt earlier in this chapter. Let us go through some other queries that can be executed on a spatial index in Solr.

The bbox query

The working of a bbox filter is similar to that of geofilt, except that the former uses the bounding box of a calculated circle. The query remains the same, except that we use the {!bbox} filter instead of the {!geofilt} filter. To convert the earlier query to bbox from geofilt, we run the following query:

http://localhost:8983/solr/collection1/select/?q=*:*&fq={!bbox pt=28.643059,77.368885 sfield=store d=10}

The output in our case would remain the same as that in the figure – Stores within 10 km from our location point – shown earlier in this chapter, but the search now includes the grayed-out area, as shown in the...

Distance sort and relevancy boost

During spatial search, it may be required to sort the search results on the basis of their distance from a specific geographical location (the lat-lon coordinate). With Solr 4.0, the spatial queries seen earlier are capable of returning a distance-based score for sorting and boosting.

Let us see an example wherein spatial filtering and sorting are applied and the distance is returned as the score simultaneously. Our query will be:

http://localhost:8983/solr/collection1/select/?fl=*,score&sort=score asc&q={!geofilt score=distance sfield=store pt=28.642815,77.368413 d=20}

The query output from Solr shows four results along with their scores. Our results are sorted in ascending order on score, which represents the distance as per our query. Hence, the results that are closest to our location appear on top.

The execution of the previous query yields the following output:

In order to add user keywords to the previous Solr query, we will have to add an additional...

Advanced concepts

As discussed earlier, RPT is based on a model where the world is divided into grid squares or cells. This is done recursively to get almost any amount of accuracy required. Each cell is indexed in Lucene with a byte string that has the parent cell's byte string as a prefix. Therefore, it is named PrefixTree. The PrefixTreeStrategy class for indexing and search uses a SpatialPrefixTree abstraction that decides how to divide the world into grid squares and what the byte encoding looks like to represent each grid square. It has two implementations, namely geohash and quadtrees. Let us look at both implementations in detail.

Quadtree

A quadtree is a simple and effective spatial indexing technique wherein each of the nodes represents a bounding box covering the parts of the space that has been indexed. Each node is either a leaf-node that contains one or more indexed points with no child, or an internal node with four children, one for each quadrant. The index structure of a quadtree...

Summary

In this chapter, we learnt how Solr can be used for geospatial search. We understood the features provided by the Solr 4 spatial module and saw how indexing and search can be executed with Solr. We discussed different types of geofilters available with Solr and performed sorting and boosting using distance as the relevancy score. We also saw some advanced concepts of geospatial operations such as quadtrees and geohash.

In the next chapter, we will learn about the problems that we normally face during the implementation of Solr in an advertising system. We will also discuss how to debug such a problem along with tweaks to further optimize the instance(s).