Chapter 4: Exploring Search Capabilities

Now that we have set the mappings and put the data inside the indices, we can start exploring the search capabilities of Elasticsearch. In this chapter, we will cover how to search using different factors: sorting, highlighting, scrolling, suggesting, counting, and deleting. These actions are the core part of Elasticsearch; ultimately, everything in Elasticsearch is about serving the query and returning good-quality results.

This chapter is divided into two parts: the first part shows how to perform an API call-related search, and the second part will look at two special query operators that are the basis for building complex queries in the upcoming chapters.

In this chapter, we will cover the following recipes:

• Executing a search
• Sorting results
• Highlighting results
• Executing a scrolling query
• Using the search_after functionality
• Returning inner hits in results
• Suggesting a correct query
• Counting matched...

Technical requirements

All the recipes in this chapter require us to prepare and populate the required indices—the online code is available in the GitHub repository(https://github.com/PacktPublishing/Elasticsearch-8.x-Cookbook). Here, you can find scripts to initialize all of the required data.

Executing a search

Elasticsearch was born as a search engine; its main purpose is to process queries and give results as quickly as possible. In this recipe, we'll learn that a search in Elasticsearch is not only limited to matching documents—it can also calculate additional information that's required to improve the quality of the search.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or something similar. I suggest using the Kibana console as it provides code completion and better character escaping.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt commands or the ch04/populate_kibana.sh script, which is available in the online code...

Sorting results

When searching for results, the standard criterion for sorting in Elasticsearch is the relevance to a text query. Often, real-world applications need to control the sorting criteria in scenarios, such as the following:

• Sorting a user by their last name and their first name
• Sorting items by stock symbols and price (ascending and descending)
• Sorting documents by size, file type, source, and more
• Sorting items related to the maximum, the minimum, or the average of some of the children fields

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute...

Highlighting results

Elasticsearch performs a good job of finding matching results in big text documents. It's useful for searching text in very large blocks. However, to improve user experience, you need to show users the abstract—a small portion of the text part of the document that has matched the query. The abstract is a common way to help users understand how the matched document is relevant to them.

The highlight functionality in Elasticsearch is designed to do this job.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands...

Executing a scrolling query

Every time a query is executed, the results are calculated and returned to the user in real time. In Elasticsearch, there is no deterministic order for records—pagination on a big block of values can lead to inconsistency between results. This is due to added and deleted documents and also documents that have the same score.

The scrolling query tries to resolve this kind of problem by giving a special cursor that allows the user to uniquely iterate all of the documents. 

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute...

Using the search_after functionality

Elasticsearch's standard pagination, using from and size, performs very poorly on large datasets because, for every query, you need to compute and discard all of the results before the from value. Scrolling doesn't have this problem, but it consumes a lot due to memory search contexts; therefore, it cannot be used for frequent user queries.

To bypass these problems, Elasticsearch 5.x, and greater, provides the search_after functionality. This provides faster skipping for scrolling results.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch...

Returning inner hits in results

In Elasticsearch, when using nested and child documents, we can have complex data models. By default, Elasticsearch only returns documents that match the searched type and not the nested or children ones that match the query.

The inner_hits function was introduced in Elasticsearch 5.x to provide this functionality.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt command, which is available in the online code.

How to do it...

Suggesting a correct query

It's very common for users to commit typing errors or to require suggestions for the words that they are writing. These issues are resolved in Elasticsearch using the suggested functionality.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt command, which is available in the online code.

How to do it...

To suggest relevant terms by querying, we will perform the following steps:

1. From the command line, we can execute...

Counting matched results

Often, it is necessary to only return the count of the matched results and not the results themselves.

There are a lot of scenarios involving counting, such as the following:

• To return the number of something (for example, the number of posts on a blog and the number of comments on a post).
• If you need to validate whether some items are available, For example, are there posts? Are there comments?

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ...

Explaining a query

When executing searches, it's very common to have documents that do match or don't match the query as expected. To easily debug these scenarios, Elasticsearch provides the explain query call. This allows you to check how the scores are computed against a document.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt command, which is available in the online code.

How to do it...

The steps that are required to execute...

Query profiling

When you are executing queries, sometimes, they are not as fast as you think. The reasons why some queries need a lot of time to be executed vary, but using tools to profile them can solve your issues more quickly.

This feature is available from Elasticsearch 5.x, or greater, via the profile API. It allows the user to track the time spent by Elasticsearch in executing a search or an aggregation.

Note

This is only a debug tool because of the significant overhead involved in its execution: for each Lucene step, it computes the part of scores, which takes a significant amount of time during query execution.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest...

Deleting by query

We saw how to delete a document in the Deleting a document recipe of Chapter 3, Basic Operations. Deleting a document is very quick, but it requires knowing the document ID for direct access and, in some cases, the routing value, too.

Elasticsearch provides a call to delete all of the documents that match a query using an additional module called re-index, which is installed by default.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana...

Updating by query

In Chapter 3, Basic Operations we saw how to update a document in the Update a document recipe.

The update_by_query API call allows the user to execute an update on all the documents that match a query. It is very useful if you need to do the following:

• Reindex a subset of your records that match a query. This is very common if you change your document mapping and need the documents to be reprocessed.
• Update the values of your records that match a query.

This is the Elasticsearch version of the SQL update command.

This functionality is provided by an additional module, called reindex, which is installed by default. 

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman...

Matching all of the documents

One of the most common queries is the match_all query. This kind of query allows the user to return all of the documents that are available in an index. The match_all query and other query operators are part of the Elasticsearch query DSL.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe in Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt command, which is available in the online code.

How to do it...

In order to execute a match_all query, we will perform the following...

Using a Boolean query

Most people who use a search engine have, at some point or another, used the minus (-) and plus (+) syntax to include or exclude query terms. A Boolean query allows the user to programmatically define queries to include, exclude, optionally include (should), or filter in the query.

This kind of query is one of the most important ones because it allows the user to aggregate a lot of simple queries or filters, as we will see in this chapter, to build a larger, more complex one.

There are two main concepts that are important in searches: query and filter. The query concept means that the matched results are scored using an internal Lucene scoring algorithm; in the filter concept, the results are matched without scoring. Because the filter doesn't need to compute the score, it is generally faster and can be cached.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch...

Using the search template

Elasticsearch provides the capability of providing a template and some parameters to fill it. This functionality is very useful because it allows you to manage the query templates stored in the .scripts index and also allows you to change them without changing the application code.

Getting ready

You will need an up-and-running Elasticsearch installation, as described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute these commands, any HTTP client can be used, such as Curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console as it provides code completion and better character escaping for Elasticsearch.

To correctly execute the following commands, you will need an index populated with the ch04/populate_kibana.txt command, which is available in the online code.

How to do it...

The template query is composed...