Aggregation Frameworks

The two key features of Elasticsearch are search and data analytics. In the previous two chapters, we learned about the search API and how to design search data modeling. We also used real-world examples from the Investor Exchange (IEX) Cloud ETF system to practice using the search feature. In this chapter, we will discuss data analytics using the aggregation framework. Aggregation can be thought of as a unit of work for building analytic information on a set of documents. The framework consists of many building blocks that can be composed to build a complex summary of the data selected by a search query. The framework is straightforward, simple, extensible, quick to access, and awesome. It can be very helpful for our business.

By the end of this chapter, we will have covered the following topics and used IEX historical ETF data to work on supported aggregation...

ETF historical data preparation

In the previous chapter, we introduced dividend data from IEX. IEX also provides historical data such as the closing price, opening price, highest price, lowest price, and closing price change. The following screenshot will show the historical price responses from IEX. We will retrieve one month of ACWF ETF data:

We list each returned field and give a short description for each one in the following table:

Field	Type	Description
date	date	The trading date
open	float	The adjusted opening price of the ETF on that date
high	float	The adjusted highest price of the ETF on that date
low	float	The adjusted lowest price of the ETF on that date
close	float	The adjusted closing price of the ETF on that date
volume	long	The adjusted daily trading volumes traded on that date
change	float	The closing price change between the trading day and...

Aggregation query syntax

The following code block in regular expression syntax demonstrates the basic structure of the aggregation query syntax:

"aggs":{
    "name_1": {
        "type": { body }
            [,"aggs" : {[sub aggregation]+}]?
    }
    [,"name_2":{...}]*
}

We can have multiple aggregations in one shot. In addition, if we have to use complex aggregation logic to solve a problem, we may use sub-aggregations. The aggs keyword is the short form of the aggregations keyword. The name_1 word is the name of the aggregation. Elasticsearch supports more than one aggregation on the same level, such as name_2. The type word is used to define the type of the aggregation such as terms, stats, and range. The body word specifies the criteria of the aggregation.

One of the powerful features of aggregations is the ability to embed aggregations...

Matrix aggregations

Matrix aggregations is a family of functions, which is still being developed to provide a way to manipulate multiple fields at the same time and generate results for the fields in a matrix form. Matrix stats is the only type supported in matrix aggregations since version 5.0.

Matrix stats

This aggregation computes the numeric statistics on a set of given document fields, as shown in the following table:

Statistics measure	Description
count	The number of samples measured.
mean	The average value of the field measured from the sample.
variance	How far the values of the field measured are spread out from its mean value. The larger the variance, the more spread from its mean value.
skewness	This...

Metrics aggregations

The metrics aggregations family provides common uses of functions to perform simple mathematical operations on values in one or more fields, and help to analyze grouped sets of documents. We will illustrate each type of aggregation in the metrics family using the fields from the cf_etf, cf_etf_hist_price, and cf_etf_dividend indexes in the following subsections.

avg

Compute the average of the numeric field value of the records. The following example aims to find the average change of the ACWF ETF in the cf_etf_hist_price index, and the value is 0.0624:

"query": { "match": { "symbol": "ACWF"}},"aggs": {"acwf_avg_close_price":{"avg": ...

Bucket aggregations

Bucket aggregation is a family that provides mechanisms for segmenting document sets into groups (buckets). Thus, each bucket is associated with a criterion, which determines whether the document in the current context falls into it. The following sections will illustrate each type of aggregation in the bucket family.

histogram

The purpose of the histogram aggregation is to group values into ranges with a fixed interval and count the documents. The starting range is 0 or a given offset. The range distance is specified by the interval value. The rounding formula is bucket_key = Math.floor((value - offset) / interval) * interval + offset. The "extended_bounds" : {"min" : lv,"max"...

Pipeline aggregations

As the name pipeline suggests, pipeline aggregations allow us to pass the result of an aggregation as the input to the aggregation in the next stage. To pass the result, a bucket_path parameter is provided to let us specify which source will be worked on in the next stage. To define the source, we need to follow this syntax:

<aggregation_name>[<aggregation_separator>,<aggregation_name>]*[<metric_separator>,<metric>]

Here aggregation_separator is >, metric_separator is ., and the metric is the name of the metric produced in the previous stage. Pipeline aggregations can be classified into two families, parent and sibling. We will illustrate each aggregation of both families in the following subsections.

Sibling family

...

Post filter on aggregations

Typically, search results and aggregation results refer to the same query. If we want to display the search results and the aggregation results in different ways—such as applying filters to display narrowed-down search results—then we can run post_filter after the query. The post_filter parameter has no effect on aggregation. For example, let's use post_filter in the example of the term bucket aggregation. We only want to retrieve the rating = 5 documents in the search result, which is at the top of the result from the term aggregation on all of the rating field values.

In the following screenshot, only the rating = 4 documents are retrieved back, and the total hits is 34 instead of the total 314 documents:

Summary

Hurrah! We have completed looking at one of the most important features in Elasticsearch. We've learned about how to perform aggregations with well-designed examples, and delved into most of the types of aggregations. We've also learned how to use IEX ETF historical data to plot a graph of different types of moving averages, including forecasted data supported by the model.

In the next chapter, we will talk about the ingest node, in which documents can be pre-processed before the actual indexing takes place. We can define a pipeline to specify a series of processors to execute in the same order as they are declared. The ingest node intercepts bulk and index requests, applies the transformation, and then passes the documents back. By default, all nodes are enabled and we can disable the functionality of the node in the configuration file.

...