Working with Elasticsearch Analysis Plugins

In the previous chapter, we learned about Elasticsearch SQL. We started with a basic understanding of the Elasticsearch SQL language from its conceptual definition to practical examples. We have written a SQL statement in command-line mode. We have played with the SQL REST API in the Kibana Dev Tools console. We also gave a direction for programming with Elasticsearch SQL through JDBC (Java Database Connectivity).

In this chapter, we will introduce the Elasticsearch plugin, which is an approach to enhance Elasticsearch's capabilities in a customized way. Elasticsearch comes with many core built-in plugins. Additionally, there are many available custom plugins that have been contributed by different communities too. In this chapter, however, we will focus only on the Analysis Plugins.

In this chapter, we will cover the following...

What are Elasticsearch plugins?

Plugins are used to extend the basic capabilities of Elasticsearch in a customized way; for example, by adding custom mapping types, custom analyzers, native scripts, custom discovery mechanisms, and more. A plugin must contain a property file, named plugin-descriptor.properties, and it may optionally contain JAR files, scripts, and configuration files. The source of plugins can be classified into two types. The first one is the built-in core plugin that is part of the Elasticsearch package and is maintained by the Elastic team. The second one is developed by individuals or different communities. For example, the ICU Analysis plugin and the Smart Chinese Analysis plugin (for more information about these plugins, refer to https://github.com/elastic/elasticsearch/tree/master/plugins) are the core plugins shipped with the product, while the IK Analysis...

Working with the ICU Analysis plugin

The ICU Analysis plugin is a set of libraries that integrates the Lucene ICU module into Elasticsearch. Essentially, the purpose of the ICU is to add the support of Unicode and globalization to provide better text segmentation analysis of Asian languages. From Elasticsearch's point of view, this plugin provides new components in text analysis, as shown in the following table:

Components		Description
Character filter	ICU Normalizer Character Filter	The icu_normalizer character filter converts text into unique, equivalent character sequences. It supports three optional parameters: name, mode, and unicode_set_filter. The name parameters can be nfc, nfkc, and nfkc_cf (the default). The mode parameter can be decompose.
Tokenizer	ICU Tokenizer	The icu_tokenizer tokenizer splits a piece of text into words on word boundaries. It adds...

Working with the Smart Chinese Analysis plugin

The Smart Chinese Analysis plugin integrates Lucene's Smart Chinese analysis module into Elasticsearch for analyzing Chinese or mixed Chinese-English text. The supported analyzer uses probability knowledge based on a hidden Markov model on a large training corpus to find the optimal word segmentation for Simplified Chinese text. The strategy it uses is to first break the input text into sentences and then perform segmentation in a sentence to obtain words. This plugin provides an analyzer, which is called the smartcn analyzer, and a tokenizer called smartcn_tokenizer. Note that both cannot be configured with any parameter.

To install the smartcn Analysis plugin in the Elasticsearch Docker container, use the commands shown in the following screenshot. We then restart the container to make the plugin effective:

...

Working with the IK Analysis plugin

The IK Analysis plugin belongs to the community contributed plugin category. It is an open source project (https://github.com/medcl/elasticsearch-analysis-ik) that provides a lightweight Chinese word segmentation toolkit based on the Java language. It integrates the Lucene IK Analyzer into Elasticsearch and supports a customized dictionary. It works with English and Chinese and is also compatible with Korean and Japanese characters. It supports hot swapping to update dictionary contents without the need to restart the Elasticsearch server. This plugin provides two types of analysis: fine-grained and coarse-grained. Fine-grained analysis includes the ik_max_word analyzer and the ik_max_word tokenizer. They will generate more tokens and they are suitable for the term query. Coarse-grained analysis includes the ik_smart analyzer and the ik_smart...

Summary

Voila! We have completed this chapter. We should now understand what Elasticsearch plugin management is and how it enhances the core functionality of Elasticsearch.

In this chapter, we introduced Analysis Plugins and deep dived into their details. We practiced the ICU Analysis plugin, the Smart Chinese Analysis plugin, and the IK Analysis plugin to perform analysis for Chinese texts. It seems that the IK Analysis plugin is better than the other two plugins due to its ability to segment word boundaries. We also added a new custom dictionary to improve word segmentation in order to make it work better.

In the next chapter, we will study the machine learning support that is available in Elasticsearch and use a Python scikit-learn package to work with Elasticsearch. We will follow two examples; the first one will create a single metric job to track the custom indicator to...