Chapter 7. Customizing Elastic Stack

One point to check whether a tool is great, is to find the answer of the following question: how much can it be customized? If you are selling your product, and if you have answer to that question like - Yes, and these are the ways. You already sold the product. This is an era of open source products, and level of customization of the product, plays a major role in deciding whether a product should be utilized for our requirement or not. There are many things your buyer would want to do with your setup, such as writing custom plugins, modifying existing things in product, modifying the UI for your branding, and much more.

In this chapter, we will learn how to customize the Elastic Stack by extending or modifying the components. The following is a list of topics that we will capture in this chapter:

• Extending Elasticsearch

• Extending Logstash

• Extending Beats

• Extending Kibana

Extending Elasticsearch

Elasticsearch is a rich-in-feature, stable, and mature project, yet it is growing everyday. Like a good open source project, Elasticsearch offers ways to extend and let users customize Elasticsearch in their own ways. There are many things you would want to achieve through customization, to name a few: you would want to support a new repository to which Elasticsearch can connect and keep data, you might be interested to write your own analyzer, which does some additional tasks as compared to the existing analyzers in Elasticsearch, and so on. For the previous versions of Elasticsearch (prior to v5), developers used to create their own rivers to support a new data source. In fact, using the HTTP and transport modules, there are numerous opportunities of extending Elasticsearch beyond Elasticsearch packaged code and providing something that helps to integrate Elasticsearch with some other tool or technology.

Although not too many attempts were made to customize Elasticsearch...

Extending Logstash

There are two ways to extend Logstash: either by modifying the core or adding one more plugin to its repository. As we have already learned, there are four types of plugins - Input, Output, Filter, and Codec. Generating plugin in Logstash is very easy and needs only one command to be run; the generated plugin needs only our custom logic for it to be implemented.

Logstash plugins are written in Ruby programming language (http://www.ruby-lang.org/en/). These plugins are actually ruby gems. To be able to write effective gems, you should be familiar with Ruby.

In this section, we will develop our own input plugin, which will fetch data from a web service. The complete code of the plugin is available at GitHub at https://github.com/kravigupta/mastering-elastic-stack-code-files/tree/5.1.1/Chapter07/logstash-input-weather. The purpose of this exercise is to learn how a plugin can be generated and customized to serve the purpose, along with understanding of structure of the plugin...

Extending Beats

As we learned through previous chapters, there are four core Beats provided by Elastic Team: Filebeat, Packetbeat, Metricbeat, and Winlogbeat. There is a huge list of community Beats as well available at https://www.elastic.co/guide/en/beats/libbeat/5.1/community-beats.html.

All of these are built on top of the libbeat framework. In case you don't find a beat that meets your expectations and you want to create a new one, your new beat will also follow the same framework. This framework is designed in such a way that all you need to take care is your custom logic, which will fetch the data from a source and prepare an event to be sent to either Elasticsearch or Logstash.

In this section, we will see how to create a beat by utilizing the libbeat framework. Beats were created to minimize the usage of Logstash. We will also use the same use case used for Logstash plugin. We will read the data from a web service and index the data to Elasticsearch. For reference, all of the generated...

Extending Kibana

When you set up Elastic Stack for your environment, you sense the needed changes in the stack to fulfill your requirements. For most of the cases, changes done at Elasticsearch are minimum as compared to rest of the components. Kibana being the face of the stack undergoes maximum changes. If nothing complex, your client (in case you are a service provider) would like to customize the UI to match their company's website look.

There are ways in which Kibana can be customized and extended-we can write Kibana plugins, create new visualizations, modify the source, and so on. To do any of these, you must be familiar with HTML, CSS, Javascript, Node.js, AngularJS, and probably relevant library to be used for charts and visualizations. Kibana is basically a Node.js app, thus knowing Node.js will help greatly to understand and customize Kibana.

In this section, we will develop a plugin that changes the appearance of the Kibana UI. For our use case, we will keep it simple and modify...

Summary

If we talk about customization of four tools, it would take four different chapters to begin with. However, we tried to understand the ways a tool can be extended. In this chapter, we learned about extending Elasticsearch by learning how a plugin is organized and popular ways to customize.

Later in the chapter, we created a Logstash plugin, which captured data from the Openweathermap.org server and sent it to Elasticsearch. We also learned about creating a new beat with example of collecting data from the same web service. Finally, we set up the Kibana development environment and learned to create a very basic hack of changing the title of the Kibana pages by appending PACKT to it.

In the next chapter, we will be revisiting Elasticsearch and its APIs. We will learn how to access the APIs and how they are useful to us. The chapter will also introduce ingest nodes which is a new addition to Elasticsearch v5.x onwards.