Speeding Up Intelligence – Data Summarization

In this chapter, we will cover the methods that exist within Splunk to speed up intelligence. You will learn about:

• Calculating an hourly count of sessions versus completed transactions
• Backfilling the number of purchases by city
• Displaying the maximum number of concurrent sessions over time

Introduction

In Chapter 5, Extending Intelligence - Datasets, Modelling and Pivoting, we learned all about data models and how they can be accelerated to facilitate faster Pivot reporting. Data model acceleration works by leveraging data summarization behind the scenes. In this chapter, we will take a look at two more data summarization methods in Splunk: summary indexing and report acceleration. These methods enable you to speed up reports or preserve focused statistics over long periods of time. You will learn how to populate summary indexes, use report acceleration, backfill summary indexes with historical data, and more.

Data summarization

Big data is just that, big, and even with the best infrastructure, it can be extremely...

Calculating an hourly count of sessions versus completed transactions

From an operational intelligence standpoint, it is interesting to understand how many visitors we have to our online store and how many of these people actually purchase something. For example, if we have 1,000 people visiting a day, and only 10 people actually purchase something, this might indicate something is not quite right. Perhaps the prices of our products are too high, or the site might be difficult to use and thus needs a redesign. This information can also be used to indicate peak purchasing times.

In this first recipe, we will leverage summary indexing to understand how many sessions we have per hour versus how many actual completed purchase transactions there have been. We will plot these on a line graph going back the last 24 hours.

...

Backfilling the number of purchases by city

In the previous recipe, you generated an hourly summary and then, after waiting for 24 hours, you were able to report on the summary data over a 24-hour period. However, what if you wanted to report over the past 30 days or even 3 months? You would have to wait a long time for your summary data to build up over time. A better way is to backfill the summary data over an earlier time period, assuming you have raw data for this time period in Splunk.

In this recipe, you will create a search that identifies the number of purchases by city on a given day, and write this search to a summary index. You will leverage the IP location database built into Splunk to obtain the city based on the IP address in the results. You will then execute a script that comes bundled with Splunk in order to backfill the summary for the previous 30 days. Following...

Displaying the maximum number of concurrent sessions over time

In the past two recipes of this chapter, you leveraged a method of data summarization called summary indexing to summarize data into a new index, which you then reported on. In this recipe, you will use another method of data summarization known as report acceleration to speed up your report times.

In this recipe, you will create a report to look for the maximum number of concurrent sessions over a time period of 30 days. This report will then be accelerated to reduce the time taken to execute the search.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time - Getting Data...