Chapter 5. Extending Intelligence – Data Models and Pivoting

In this chapter, we will introduce the Splunk data model and pivoting functionality. We will cover the following recipes:

• Creating a data model for web access logs

• Creating a data model for application logs

• Accelerating data models

• Pivoting total sales transactions

• Pivoting purchases by geographical location

• Pivoting slowest responding web pages

• Pivot charting top error codes

Introduction

In many of the previous chapters, we leveraged Splunk's SPL quite a bit in order to build searches, reports, and dashboards. In this chapter, we will learn how to leverage Splunk's data model and Pivot functionality, and demonstrate how these can be leveraged by less technical users to easily build reports, charts, and dashboards.

The first set of recipes in this chapter involves building Splunk data models. Data models allow Splunk datasets to be mapped, together with associated knowledge, into a hierarchical structure that encapsulates a number of Splunk searches behind the scenes. These models power Splunk's Pivot tool and allow the users to create dynamic reports and dashboards, without the need to write any searches. Data models are somewhat analogous to relational database schemas; in that, they present data to Pivot as rows and columns.

Data models are typically built by individuals who are familiar with Splunk's SPL using the Data Model Editor. Data models have a hierarchical...

Creating a data model for web access logs

In this first recipe, you will create a data model for our web access logs. You will use Splunk's Data Model Editor to do this and define a number of object types, and add constraints and attributes.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should also have completed the recipes from the earlier chapters. You should be familiar with navigating the Splunk user interface.

How to do it…

Follow the steps in this recipe to create the Web Access data model:

1. Log in to your Splunk server.

2. Select the Operational Intelligence application.

3. Select the Settings menu item at the top-right corner of the screen, and then select Data models.

   

4. An empty list of data models to manage will load. Click on the New Data Model button located at the top-right corner of the screen.

   

5. A pop-up box will be displayed. Enter Web Access in the Title field and as...

Creating a data model for application logs

This recipe is similar to the first, except this time you will create a data model for application logs. You will use Splunk's Data Model Editor to do this and will define a number of object types, and add constraints and attributes. In order to save pages, this recipe will be lighter on screenshots than the first recipe. The first recipe should therefore be used as a reference where needed.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should have also completed the recipes from the earlier chapters. You should have also completed the first recipe in this chapter and be familiar with navigating the Splunk user interface.

How to do it…

Follow the steps in this recipe to create the Application data model:

1. Log in to your Splunk server.

2. Select the Operational Intelligence application.

3. Select the Settings menu item at the top-right corner...

Accelerating data models

Splunk has several options for optimizing search performance, including summary indexing, report acceleration, and data model acceleration. We will cover both summary indexing and report acceleration later in this book. Data model acceleration helps to speed up reporting for the Object Attributes defined in a data model. This acceleration can then be leveraged by the Pivot tool when reporting.

In this recipe, you will accelerate the data models that we just created in order to familiarize yourself with the process and enhance understanding. Ordinarily, you would only really want to use data model acceleration for reporting on extremely large datasets over a period of time.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In, and the completed data model recipes from earlier in this chapter. You should be familiar with navigating the Splunk user interface...

Pivoting total sales transactions

Now that we have built a couple of data models, we can begin using Splunk's Pivot tool to search and report the data without needing to write any searches.

In this recipe, you will start to get familiarized with the Pivot interface and use it to calculate total sales transaction data. You will focus on identifying successful checkout transactions. These are important from an intelligence standpoint, as they indicate that a sale has occurred and payment has been made successfully. This data will then be populated on the Product Monitoring dashboard. You will be using the transaction data model object that we defined in the Application data model.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In, and the completed recipes from earlier in this chapter. You should be familiar with navigating the Splunk user interface.

How to do it…

Follow these steps...

Pivoting purchases by geographic location

In the previous recipe, you performed a simple count of the number of successful sales transactions. In this recipe, we will get an insight into these sales by exploring where in the world sales requests are coming from. To do this, you will leverage the built-in geolocational abilities of Splunk. Firstly, you will amend the Application data model to bring in geolocational object attributes. Then you will pivot this data to map purchases by location.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In, and the completed recipes from earlier in this chapter. You should be familiar with navigating the Splunk user interface.

How to do it…

Follow these steps in this recipe to pivot purchases by geographic region:

1. Log in to your Splunk server.

2. Select the Operational Intelligence application.

3. Select the Settings menu item at the top-right corner of...

Pivoting slowest responding web pages

In the past couple of recipes, we worked with the Application data model and added some additional reports to the Product Monitoring dashboard related to sales and customer location. In the next couple of recipes, you will look at the operational health of our environment and begin creating an Operational Monitoring dashboard.

The response time of a web application is one of the most important factors in determining overall user experience, and high response times could lead to lost customers, who are not prepared to deal with slowly loading web pages.

In this recipe, you will use the Pivot tool to table the response times for the various web pages in our web application and identify the pages that are taking the longest to load. You will add this report to a new Operational Monitoring dashboard.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data...

Pivot charting top error codes

In this final recipe, you will use the Pivot tool to chart the top error codes over time. Error codes from web application logs generally fall into two main categories: client-side errors and server-side errors. Plotting the error codes over time will help identify which errors are occurring, the types of errors, and when they occur.

While this recipe is slightly less technical to implement than the previous recipe, it will serve to reinforce understanding and less instruction will be provided.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In, and the completed recipes from earlier in this chapter. You should be familiar with navigating the Splunk user interface.

How to do it…

Follow these steps to pivot chart the top error codes:

1. Log in to your Splunk server.

2. Select the Operational Intelligence application.

3. Select the Pivot menu item from the application...