Chapter 6. Graph Exploration in Kibana

In earlier chapters, we went through a logging and metric analytics scenario that mainly leveraged the aggregation API in Elasticsearch.

But what if the need was not necessarily to highlight the KPI in the data, but rather to show the interconnection of data based on relevancy? This is where Elastic Graph comes into play. Graph is an X-Pack plugin that allows us to reveal significant connections between data indexed in Elasticsearch.

Elastic Graph comes with a new API in Elasticsearch and a new UI in Kibana, which offers a totally different approach to exploring data: rather than addressing data through the angle of value aggregation and narrowing them down by filtering to discover patterns, Graph allows you to play with vertices (the terms indexed in Elasticsearch) and connections (how many documents share the terms in the index) and map out significant relations.

In this chapter, we'll learn the following concepts of the Elastic Graph flow:

• Differentiating...

Introducing the basics of Elastic Graph

Elastic Graph was created to reveal significant relations between data, so that we can see how the variables in question interact. It forms recommendations based on these relations. Data is highly connected, either implicitly or explicitly. These connections can be represented as a graph. Graph based data analysis provides unique insights based on the use case:

• In a search use case, using a graph, the search experience could be enhanced if the user gets related content based on the query they submitted. This is typically what we could see on an e-commerce website; for example, when purchasing a phone, you could get related accessories. But in the context of Elasticsearch, based on the click stream on a website, the user could get real-time, relevant, and significant suggestions based on his purchase behavior.

• In the security analytics use case, suspicious connections could be proactively detected based on the logged data. If we have all the access logs...

Exploring the Stack Overflow dataset with Elastic Graph

Stack Overflow is a website widely used to ask and answer questions about a very large set of topics in the computer science industry. It's a perfect resource to try out Elastic Graph, as the data it holds will contain users who are connected to questions, answers, tags, comments, and so on. In this section, we'll index the Stack Overflow dataset in Elasticsearch, look at the structure of the data, and build relations using Elastic Graph.

Prepare to graph!

The dataset we will use is located in the source attached to this book in the Chapter 6 folder. You will find a ZIP file called StackOverflow4Graph.zip that contains the following files:

• IndexPosts.py : Python script that indexes the data in your Elasticsearch cluster

• Posts.csv: The dataset itself

• readme.txt: The readme file, which, by the way, contains a link to a tweet that illustrates what we are going to do in this part

The following example gives an idea of Stack Overflow Graph...

Summary

In this chapter, we have seen the difference between industry Graph technologies and what Elastic Graph provides. Elastic Graph ties up aggregation and relevancy features to connect the document in Elasticsearch. Through this process, users get a way to build recommendations out of the indexed content. We then looked at some examples that illustrate how Elastic Graph can be used through simple exploration up to advanced exploration with multi-term graphs and advanced drill-downs.

In the next chapter, we'll start to delve fairly deeply into the world of Kibana customization by implementing an extension for Timelion.