Chapter 5. Customizing Hunk

Splunk has a rich SDK that allows you to create custom dashboards with extended functionality. The aim of this chapter is to show you how to create a custom dashboard using data stored in Hadoop. We are going to visualize data containing aggregated customer data records on Google Maps. The goal is to show a customer activity heatmap.

What we are going to do with the Splunk SDK

Splunk has various SDKs for different languages and platforms. We are going to talk about them shortly and won't cover deep application development. Our goal is to get the result as rapidly as possible and using few application development skills. We can always turn to custom development, and our goal is to get quick insights as soon as possible.

There is a nice portal—https://splunkbase.splunk.com—where you can find hundreds of published applications. Extending Splunk with a SDK is not something special; it's a recommended approach to get extended functionality. This book has a chapter describing integration with MongoDB. The MongoDB integration application is taken from the splunkbase portal. We encourage you to look through published apps before starting to develop your own.

Supported languages

You can use several languages to create custom applications for Splunk:

• Python

• Java

• JavaScript

• PHP

• Ruby

• C#

Great, we have a set of popular languages; it shouldn...

Dashboard customization using Splunk Web Framework

We have discovered many different dashboards created using the default Hunk functionality. Now it's time to create our own dashboard with unique functionality using the Splunk JS SDK API (JavaScript).

Functionality

We are going to use the Splunk JS API to customize our map visualization and create input controls.

We will cover these functions:

• Extending an existing map component and making it display rectangles

• Configuring input controls to modify search queries and display different data dimensions: incoming, outgoing SMS activity, and so on

A description of time-series aggregated CDR data

We used the Oozie coordinator in Chapter 1, Meet Hunk, to import massive amounts of data. Data is partitioned by date and stored in binary format with a schema. It looks like a production-ready approach. Avro is pretty well supported across the whole Hadoop ecosystem. Now we are going to create a custom application using that data. Have a look at the description of the data.

Here is a description of the data stored in the base table:

• Square ID: The ID of the square that is part of the Milano grid type: numeric.

• Time interval: The beginning of the time interval expressed as the number of milliseconds elapsed from the Unix Epoch on January 1, 1970 at UTC. The end of the time interval can be obtained by adding 600,000 milliseconds (10 minutes) to this value.

• Country code: The phone code of a nation. Depending on the measured activity this value assumes different meanings that are explained later.

• SMS-in activity: The activity in terms of received...

Implementation

Now it's time to implement heatmap application. We will start from creating query to get sample data for application and then move forward to coding visualization using Javascript and Python.

Querying the visualization

Let us start to get acquainted with the data we have. We are going to explore sample data to make the process faster.

We are going to use the next query during development. We will display a heatmap for the center of Milano. The other simplification is a hardcoded time interval. We removed all other intervals from the sample dataset using the Pig script earlier. The general idea is to reduce the amount of data and make the development cycle shorter:

(index="milano_cdr_sample" time_interval=1385884800000 
AND (
 (square_id >5540 AND square_id < 5560) OR 
 (square_id >5640 AND square_id < 5660) OR 
 (square_id >5740 AND square_id < 5760) 
 )
)

| fields square_id, sms_in, time_interval 

| stats sum(sms_in) as cdrActivityValue by square_id, time_interval...

Custom map components

We change the behavior of the GoogleMapView component:

var customHeatMap = GoogleMapView.extend

The idea is to override the render method and provide custom handling for the search result. Each row returned from the search manager is presented as:

google.maps.Rectangle

Other components

The SearchManager component is used to submit the search query and retrieve the result from Splunk:

var search = new SearchManager

This drop-down input control with predefined values provides access to different types of activity:

new DropdownView

By the way, we can use a second search manager and populate DropdownView content dynamically with the search result.

The final result

You can see a part of Milano highlighted with a heatmap:

The heatmap displays reasonable data. Low activity is observed in the park areas and high activity near metro stations.

Summary

Splunk provides various approaches for custom application development. It's possible to start with a so-called simple XML that provides basic functionality. XML declarations allow us to create custom input forms and even visualizations on Google Maps. When you need more, you can turn to classical application development using various languages. The earlier example explained one approach to extending existing components. With a few lines of custom code we got a simple interactive map based on data stored in Hadoop and presented as a virtual index.