Building Analytics RESTful Services

So far, we have learned about basic features such as mapping, using the document analyzer, indexing, searching, and aggregation. We have also learned about advanced features such as Elasticsearch SQL, Elasticsearch Machine Learning (ML) jobs, and Elasticsearch-Hadoop (ES-Hadoop) for Apache Spark. In addition, we have studied how to write the programs with the Java high-level REST client and Spring Boot. In this chapter, we'll put these materials together to build an end-to-end real-world final project to help the readers understand how they fit together. We'll reuse some of the codes provided earlier and glue them together. This project provides a search analytics REST service powered by Elasticsearch. The data flow will be retrieved from Elasticsearch using the REST client. The data is then used to build a k-means clustering model...

Building a RESTful web service with Spring Boot

In this project, we want to view the Bollinger Band on the Kibana Visualize page. There may be more than one way to do it. What we are going to do is pre-compute the related values of the Bollinger Band and index them when the trading price of the targeted symbol is ready. In addition to the fields from the cf_etf_history_price index, we are going to index fields such as the standard deviation and the moving average of 20 trading days, the upper and lower bounds of the Bollinger Band, the typical price = (high+low+close)/3, and the predicted anomaly class from the k-means model. We will create a Bollinger Band visualization with Kibana, as shown in the following screenshot:

There are five lines on the chart. The uppermost line on the chart represents the predicted class that uses the k-means model for the 20-day standard deviation...

Integration with the Bollinger Band

In the last section, we introduced the workflow of buildAnalyticsModel. After we insert/update the ETF data to the cf_etf_history_data index, we compute the data for the Bollinger Band. Let's recall the details to compute the Bollinger Band from the Operational data analytics section of Chapter 10, Using Elasticsearch for Exploratory Data Analysis, to work on Java programming. The step-by-step instructions are as follows:

1. Collect all the related documents by performing a search operation: symbol and period are given by the user. startDate and endDate can be derived from the period. We have learned how to use Elasticsearch's high-level REST client to build a SearchRequest object in the Java high-level REST client section in Chapter 11, Elasticsearch from Java Programming. The following code block is extracted from the getBollingerBand...

Building a Java Spark ML module for k-means anomaly detection

According to the Spark MLlib guide (see https://spark.apache.org/docs/latest/ml-guide.html), starting from Spark 2.0, the RDD-based APIs in the spark.mllib package will be retired. Users should use the DataFrame-based ML API in the spark.ml package. In this project, we import several classes from this new library to build the anomaly detection model. The following code block shows a few lines from the AnomalyDetection class in the com.example.esanalytics.spark.mllib package:

import org.apache.spark.ml.clustering.KMeansModel;
import org.apache.spark.ml.feature.VectorAssembler;
import org.apache.spark.ml.clustering.KMeans;

The following diagram helps us to learn about the steps to build the model within the scope of ES-Hadoop, Spark SQL, and Spark MLlib:

The step-by-step instructions are as follows:

1. There are two major...

Testing Analytics RESTful services

We are going to present the testing of the build-analytics-model and get-register-symbols APIs from the analytic controller. The step-by-step instructions are described in the following subsection.

Testing the build-analytics-model API

This API builds the analytics model based on the user's input, such as the symbol, period, and fields to analyze. We will use rfem for the symbol, 1 year (1y) for the past period, and the 20 days standard deviation (tDStDev) as input. Please follow these step-by-step instructions:

1. Use a web browser and type http://localhost:10010/swagger-ui.html#/analytics-controller/buildAnalyticsModelUsingPOST to reach the Swagger UI page for the build-analytics-model...

Working with Kibana to visualize the analytics results

Ensure that Kibana is running before we perform the following step-by-step instructions. We are going to draw the Bollinger Band on the Kibana Visualize page:

1. Similar to the Creating index pattern subsection of Chapter 16, Machine Learning with Elasticsearch, let's create a Kibana index pattern named cf_etf_history_data, as shown in the following screenshot:

2. Use a web browser and type http://localhost:5601/app/kibana#/visualize?_g=() to reach the Kibana Visualize page. We will see a panel that is similar to the following screenshot:

3. Click the + button cover, which is by the search box, to get into the New Visualization panel, as shown in the following screenshot:

4. Click the Line button to create a line chart and Kibana will pop up a new panel to let us select the index pattern for the new line input. Select...

Summary

Woohoo! We have completed the final chapter. We glued together the knowledge we have learned and presented the result on the Kibana Visualize page. We believe that it is not easy to digest all the materials from this book. However, we are very confident that this book opens the way for readers who start out as beginners and quickly become skilled users. We have covered many advanced topics, such as Elasticsearch SQL, ES-Hadoop, ML, File Beat-Logstash-Elasticsearch-Kibana integration, and the Analytics plugin. Besides this, we have used the two most popular programming languages, Java and Python, to show the implementation of integrating Elasticsearch to build analytics applications. We hope that all readers will achieve great success in their careers with Elastic Stack.