These are the instructions you need to follow to install Splunk on your Windows desktop. Take your time and do not rush the installation. Many chapters in this book will rely on these steps:

Logging in the first time

Launch the application the first time in your default browser. You can also manually access the Splunk web page via the http://localhost:8000 URL.

Note

Splunk requires you to use a modern browser. It supports most versions of Google Chrome, Firefox, and newer versions of Internet Explorer. It may not support older versions of Internet Explorer.

Log in with the default username and password (admin : changeme) as indicated in the following screenshot:

The next step is to change the default administrator password, while keeping the default username. Do not skip this step. Make security an integral part of your day-to-day routine. Choose a password that will be secure:

Assuming all goes well, you will now see the default Splunk Search & Reporting dashboard:

It is good practice to create a custom Splunk app to isolate all the changes you make in Splunk. You may never have created an app before, but you will quickly see it is not very difficult. Here we will create a basic app called Destinations that we will use throughout this book:

We will use this bare bones app to complete the exercises in this book, but first we need to make a few important changes:

These steps will ensure that the application will be accessible to the Eventgen add-on that will be installed later in the chapter. Use the following screenshot as a guide:

Splunk permissions are always composed of three columns: Roles, Read, and Write. A role refers to certain authorizations or permissions that can be taken on by a user. Selecting Read for a particular role grants the set of users in the role permission to view the object. Selecting Write will allow the set of users to modify the object. In the preceding screenshot, everyone (all users) will have access to view the Destinations app, but only the admin (you) and a power user can modify it.

Machine data is the information produced by the many functions carried out by computers and other mechanical machines. If you work in an environment that is rich in machine data, you will most likely have many sources of readily-available machine inputs for Splunk. However, to facilitate learning in this book, we will use a Splunk add-on called the Splunk Eventgen to easily build real-time and randomized web log data. This is the type of data that would be produced by a web-based e-commerce company.

Here's an important tip. Make it a habit to always launch your command prompt in Administrator mode. This allows you to use commands that are unhindered by Windows security:

There are several different ways to stop, start, or restart Splunk. The easiest way is to do it from the web interface, as demonstrated in the preceding section. The web interface, however, only allows you to restart your Splunk instance. It does not offer any other control options.

In Windows, you can also control Splunk through the Splunkd Service as shown in the following screenshot. The d in the service name, denoting daemon, means a background process. Note that the second service, splunkweb, is not running. Do not try to start splunkweb as it is deprecated and is only there for legacy purposes. The Splunk web application is now bundled in Splunkd Service:

The best way to control Splunk is by using the command-line interface (CLI). It may require a little effort to do it, but using the CLI is an essential skill to learn. Remember to always use command prompts in Administrator mode.

In the console or command prompt, type in the following command and hit Enter on your keyboard:

C:\> cd \Splunk\bin

Here cd is a command that means change directory.

While in the C:\Splunk\bin directory, issue the following command to restart Splunk:

C:\> C:\Splunk\bin> splunk restart 

After issuing this command, splunkd will go through its restart process. Here are the other basic parameters that you can pass to the Splunk application to control Splunk:

Doing this in the console gives the added benefit of verbose messages. A verbose message is a message with a lot of information in it. Such messages can be useful for making sure the system is working correctly or troubleshooting any errors.

A successful restart of splunkd has the following output (which may vary):

We are almost there. Proceed by first downloading the exercise materials that will be used in this book. Open an Administrator command prompt and make sure you are in the root of the C: drive. If you are using Git, clone the entire project with this Git command:

C:\> git clone https://github.com/ericksond/splunk-essentials.git

You can alternatively just download the ZIP file and extract it in your computer using https://github.com/ericksond/splunk-essentials/archive/master.zip.

The Eventgen configuration you will need for the exercises in this book has been packaged and is ready to go. We are not going into the details of how to configure Eventgen. If you are interested in learning more about Eventgen, visit the project page at http://github.com/splunk/eventgen.

Follow these instructions to proceed:

Next we will see our Destinations app in action! Remember that we have configured it to draw events from a prototype web company. That is what we did when we set it up to work with Eventgen. Now let's look at some of our data:

Examine the event data that your new app is enabling to come into Splunk. You will see a lot of references to browsers, systems, and so forth: the kinds of information that make a web-based e-commerce company run.

Try changing the time range to Real-time (5 minute window) to see the data flow in before your eyes:

Congratulations! You now have real-time web log data that we can use in subsequent chapters.

Now that we have data ingested, it is time to use it in order to derive something meaningful out of it. You are still in the Destinations app, correct? We will show you the basic routine when creating new dashboards and dashboard panels.

Copy and paste the following search query in the Search Field, then hit Enter:

SPL> index=main /booking/confirmation earliest=-24h@h | timechart 
     count span=15m

After the search results render, click on the Visualization tab. This will switch your view into visualization so you can readily see how your data will look. By default, it should already be using the Column Chart as shown in the following screenshot. If it does not, then use the screenshot as a guide on how to set it:

Now that you can see your Column Chart, it is time to save it as a dashboard. Click on Save As in the upper-right corner of the page, then select Dashboard Panel as shown in the following screenshot:

Now let's fill up that dashboard panel information, as seen in the following screenshot. Make sure to select the Shared in App in the Dashboard Permissions section:

Finish up by clicking View Dashboard in the next prompt:

You have created your very first Splunk dashboard with a panel that tells you the number of confirmed bookings in the last 24 hours at 15-minute intervals. Time to show it to your boss!

Take that well-deserved coffee break. You now have a fully-functional Splunk installation with live data. Leave Splunk running for 2 hours or so. After a few hours, you can stop Splunk if you need to rest for a bit to suppress indexing and restart it when you're ready to proceed into the next chapters. Do you recall how to control Splunk from the command line?

C:\>  C:\Splunk\bin> splunk stop
C:\Splunk\bin> splunk start

In this chapter, you learned a number of basic Splunk concepts that you need to get started with this powerful tool. You learned how to install Splunk and configure a new Splunk app. You ran a simple search to ensure that the application is functional. You then installed a Splunk add-on called Eventgen, which you used to populate dummy data into Splunk in real time. You were shown how to control Splunk using the web user interface and the command-line interface. Finally, you created your very first Splunk dashboard. Now we will go on in Chapter 2, Bringing in Data, to learn more about how to input data.

Splunk was invented as a way to keep track of and analyze machine data coming from a variety of computerized systems. It is a powerful platform for doing just that. But since its invention, it has been used for a myriad of different data types, including machine data, log data (which is a type of machine data), and social media data. The various types of data that Splunk is often used for are explained in the next few sections.

Indexes are where Splunk Enterprise stores all the data it has processed. It is essentially a collection of databases that is, by default, located at $SPLUNK_HOME/var/lib/splunk. Before data can be searched, it needs to be indexed, a process we describe here.

There are two ways to create an index, through the Splunk portal or by creating an indexes.conf file. You will be shown here how to create an index using the Splunk portal, but you should realize that when you do that, it simply generates an indexes.conf file.

You will be creating an index called wineventlogs to store Windows Event Logs. To do this, take the following steps:

You will now see the new index in the list as shown here:

The preceding steps have created a new indexes.conf file. Now go ahead and inspect this file. In your administrator command prompt (assuming that you are at the root of the C: drive), type the following command:

C:\> notepad c:\splunk\etc\apps\destinations\local\indexes.conf

Every index has specific settings of its own. Here is how your index looks when automatically configured by the portal. In production environments, this is how Splunk administrators manage the indexes. Note that the max size value of 100 that you specified is also indicated in the configuration.

[wineventlogs] 
coldPath = $SPLUNK_DB\wineventlogs\colddb 
homePath = $SPLUNK_DB\wineventlogs\db 
maxTotalDataSizeMB = 100 
thawedPath = $SPLUNK_DB\wineventlogs\thaweddb 

The complete indexes.conf documentation can be found at http://docs.splunk.com/Documentation/Splunk/latest/admin/indexesconf.

You may have noticed that there is a certain pattern in this configuration file, in which folders are broken into three locations: coldPath, homePath, and thawedPath. This is a very important concept in Splunk. An index contains compressed raw data and associated index files that can be spread out into age-designated directories. Each piece of this index directory is called a bucket.

A bucket moves through several stages as it ages. In general, as your data gets older (think colder) in the system, it is pushed to the next bucket. And, as you can see in the following list, the thawed bucket contains data that has been resurrected from an archive. Here is a breakdown of the buckets in relation to each other:

Now going back to the indexes.conf file, realize that the homePath will contain the hot and warm buckets, the coldPath will contain the cold bucket, and the thawedPath will contain any restored data from the archive. This means you can put buckets in different locations to efficiently manage disk space.

In production environments, Splunk admins never use the portal to generate indexes. They make their changes in the indexes.conf file. It is best practice to always use a temporary index for new data that you are unfamiliar with. Go ahead and create a new index stanza in indexes.conf with the following information:

[temp] 
coldPath = $SPLUNK_DB\temp\colddb 
homePath = $SPLUNK_DB\temp\db 
maxTotalDataSizeMB = 100 
thawedPath = $SPLUNK_DB\temp\thaweddb 

As you may have noticed, any configuration you make in the Splunk portal corresponds to a *.conf file written to the disk. The same goes for the creation of data inputs; it creates a file called inputs.conf. Now that you have an index to store your machine's Windows Event Logs, let us go ahead and create a data input for it, with the following steps:

Before we proceed further, let's make sure that the data input works. To do so, follow these steps:

We have introduced a new concept here called  source types. A source type is a type of classification of data that has been automatically made for you when you created the data input through the Splunk portal. In the same index, due to the steps we just took precedingly, Splunk internally classified the System and Applications event logs so you can access the events separately. There will be more about source types in Chapter 3, Search Processing Language.

Go ahead and inspect the inputs.conf file:

You can add directly into the inputs.conf file by following the same format. Use the temp index for testing. Here is an example input file based on a log file that is generated by the Splunk application. Note that this is going to be redundant to what is already being ingested through the _internal index and will only serve as an example.

[monitor://C:\Splunk\var\log\splunk\eventgen.log] 
disabled = 0 
sourcetype = eventgen 
index = temp 

Changes to the inputs.conf may require a Splunk restart. You can access that information with the following search command:

SPL> index=temp sourcetype=eventgen

The complete documentation for the inputs.conf  file can be found at https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf.

If you closely followed the instructions in this book, you should now have exactly four data sources in your very own Splunk system that will be used in the remainder of the book. This is how you can query raw data from these data sources.

All throughout this chapter you have been running Splunk search queries that have returned data. It is important to understand what events and fields are before we go any further, for an understanding of these is essential to comprehending what happens when you run Splunk on the data.

In Splunk, a single piece of data is known as an event and is like a record, such as a log file or other type of input data. An event can have many different attributes or fields or just a few attributes or fields. When you run a successful search query, you will see that it brings up events from the data source. If you are looking at live streaming data, events can come in very quickly through Splunk.

Every event is given a number of default fields. For a complete listing, go to http://docs.splunk.com/Documentation/Splunk/6.3.2/Data/Aboutdefaultfields. We will now go through some of these default fields.

These default fields are name/value pairings that are added to events when Splunk indexes data. Think of fields as a quick way to categorize and group events. Fields are the primary constituents of all search queries. In later chapters you will learn more about fields and how to create custom fields from events.

Most raw data that you will encounter will have some form of structure. Just like a CSV (comma-separated value file) or a web log file, it is assumed that each entry in the log corresponds to some sort of format. Splunk 6.3+ makes custom field extraction very easy, especially for delimited files. Let's take the case of our Eventgen data and look at the following example. If you look closely, the _raw data is actually delimited by white spaces:

2016-01-21 21:19:20:013632 130.253.37.97 GET /home - 80 - 10.2.1.33 "Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J3 Safari/6533.18.5" 200 0 0 186 3804 

Since there is a distinct separation of fields in this data, we can use Splunk's out-of-the-box field extraction tool to automatically classify these fields. In your Destinations app Search page, run the following search command:

 SPL> index=main sourcetype=access_custom

This sourcetype access_custom refers to a type of file format that is generated by a server as it creates a web log file. After the data populates, click on the Extract New Fields link in the left column of the page as shown in the following screenshot:

In the resulting Extract Fields page, select one of the events that is shown in the _raw events area. Try to select an entry with the longest text. As soon as you do this, the text will appear highlighted at the top of the page, as per the following screenshot:

Click on the Next button to proceed. In the page that appears, click on the Delimiters icon as indicated in the following screenshot:

Click on Next. On the next page, click on the Comma delimiter as shown in the following screenshot:

As soon as you select the Comma delimiter, Splunk will automatically allow you to modify each field and add a label to it. Click on the pencil icon for each field to input the label. When you're done, click on the Rename Field icon. For example, to edit field3, use the following screenshot:

Do the same for the remaining fields using the following guide. These fields will be needed in future chapters. You can skip those that are not included in the following list:

When you have completed the preceding task, click on Next to proceed. In the next window, label the Extractions Name as eventgen and select the All apps permission type. Refer to the following screenshot:

Click on Finish to complete the process. Now that you have extracted new fields, these will now be readily available in your search queries as exemplified below. In the next chapter, you will be shown how to use these new fields to filter search results. An example of the kind of search you can do on them is shown here:

SPL> index=main | top http_uri

If you want to go ahead and try this out now, just to prove that you have already made changes that will help you to understand the data you are bringing in, be our guest!

In this chapter, we learned about some terms that need to be understood about big data, such as what the terms streaming data, data latency, and data sparseness mean. We also covered the types of data that can be brought into Splunk. Then we studied what an index is, made an index for our data, and put in data from our Destinations app. We talked about what fields and events are. And finally, we saw how to extract fields from events and name them so that they can be more useful to us. In the chapters to come, we'll learn more about these important features of Splunk.

Every time you execute a search, always be aware that you are running a query against a set of data that is bound by date and time. The time-range picker is on the right side of the search bar. Splunk comes with predetermined time modifiers, as seen in the following screenshot. You can also use the time-range picker to set up a custom date/time range or other advanced ranges (https://docs.splunk.com/Splexicon:Timerangepicker):

There are two types of time modifier: real-time and relative. In the preceding screenshot, the predetermined real-time modifiers are in the leftmost column, and the relative time modifiers are in the middle column.

Real-time modifiers mean that Splunk will run an ongoing, real-time search based on the specified time. For example, a real-time search that is in a 5-minute window will continuously display data within the last five minutes. If new data comes in, it will push out the oldest event within the time frame.

Relative time modifiers are just that; they collect data based on relative time measures, and will find data within the specified timeframe.

What you do not see when you are using the time modifier drop-down is that in the background, Splunk is defining the earliest time and the latest time in specific variables.

The last 15 minutes preset, for example, is equivalent to this SPL modifier:

SPL> earliest=-15m latest=now

The presets built into Splunk automatically insert the latest=now modifier. Run this search command in your Destinations app Search bar:

SPL> index=main earliest=-30m latest=now | timechart count span=5m

Notice that even if you have not changed the time modifier selected in the drop-down menu (which will not change unless you use it), the data will show that your earliest event is 30 minutes ago and your last data is recent. In other words, what you put in the search bar overrides the time modifier drop-down menu.

You can use a number of alternative ways to identify each of the time units; the most commonly supported time units listed by Splunk are:

Splunk is currently one of the best enterprise search engines, that is, a search engine that can serve the needs of any size organization currently on the market. Using a search command, you can filter your results using key phrases just the way you would with a Google search. Here are some examples for you to try out:

SPL> index=main /booking/confirmation

The preceding filters search results and only shows those with /booking/confirmation in the _raw data.

You may also add further filters by adding another phrase. It is very important to note, however, that by default, Splunk will assume that your phrases are logically chained based on an AND operator. For example:

SPL> index=main /booking 200

The preceding line of code is equivalent to the following:

SPL> index=main /booking AND 200

Similarly, you can use the OR operator to find data based on multiple filters. The following command will return all events with /booking or /destinations in the text. It is important to remember that an OR operator will always give you at least as many (or more) events than an AND operator, and AND is the default operator:

SPL> index=main /booking OR /destinations

Like any mathematical operation, you may also use parentheses to group conditions:

SPL> index=main (/booking OR /destinations) AND 200

If you have a phrase that has white space in it, it is best practice to enclose it with quotation marks, as seen in the following example:

SPL> index=main "iPhone OS"

You may also filter search results using fields. Fields are case-sensitive and a search using a specified field is generally faster than a full text search. Filtering using fields will only work if there is a defined field. In Chapter 2, Bringing in Data, you extracted new fields from the eventgen data source. Let's use that now to filter search results using custom fields:

SPL> index=main method=GET url="/home"

You can use this concept to combine data even if it comes from different indexes. Here is a query that shows the concepts we have discussed so far. Although the following query will result in a data set that will not make any sense, we include it to show how to combine data from different indexes:

SPL> (index=wineventlogs OR index=main) LogName=Application OR /booking

The most common use of the stats command is to get a count of the total number of events that are the product of a search. To see how this works, run the following search query. Notice that the pipe that precedes the stats command filters the data that will be included in the final count:

SPL> index=main earliest=-30m latest=now | stats count

The preceding query will result in a single number that represents the total of all events within the given time modifier. Change the time modifier and the number should be reduced:

SPL> index=main earliest=-15m latest=now | stats count

You may be wondering where the count came from. The true format of a stats command is stats function(X). This asks the system to return the result of the function based on the field X. When the function count is used without parentheses, Splunk assumes that you are looking for the count of all events in the given search.

The stats command becomes a very powerful tool especially when you need to group counts by fields. Here is an example:

SPL> index=main  | stats count by method

This will result in two rows of data that will show the counts of the GET and the POST methods, as shown in the following screenshot. These are two methods that are used in HTTP (website communication rules for client and server) to ask for information (GET) and submit data (POST):

You can also use the avg(X) function to get the average value of all the events based on URLs. Here is an example that you can use:

SPL> index=main | stats count by url | stats avg(count)

Some of the widely used stats functions are:

A quick way to get a summarized table based on fields is by using the top and rare commands. Run this search command:

SPL> index=main | top url

Notice that the result automatically grouped the URLs by count, calculated the percentage of each row against the whole data set, and sorted them by count in descending order. You can see a sample result in the following screenshot:

You may further tweak this search command by adding command options such as limit and showperc. Say, for example, you only want to see the top five URLs but you do not want to see the percent column. This is the command to achieve that:

SPL> index=main | top url limit=5 showperc=false

Now try the same commands, but use rare instead of top. The term rare will find those events that are the most unlikely ones. This can be a useful qualifier to use for determining outliers or unusual cases that may be due to data entry errors.

The chart command is an aggregation command that provides output in tabular or chartable format. It is a very important command that is used for many different types of visualization. Notice that if you run the following search query, it is identical to the output of the stats command:

SPL> index=main | chart count by method

For all basic purposes, you can use stats and chart interchangeably. However, there will be differences in how stats and chart group data together. It will be up to you to determine which one is your intended result. To show the differences, here are some examples:

SPL> index=main | stats count by method url

SPL> index=main | chart count by method url

The timechart command, on the other hand, creates a time series chart with statistical aggregation of the indicated fields. This command is widely used when creating different types of chart. The most common use of timechart is for examining the trends of metrics over time for visualizations including line charts, column charts, bar charts, and area charts, among others:

SPL> index=main earliest=-4h latest=now | timechart span=15m count by url 

An important option that is part of the timechart command is span. The span essentially determines how it will bucket the data based on time slices.  span=15m means it will aggregate the data and slice it every 15 minutes.

The statistical result of the command looks like this:

Although admittedly the preceding data looks dull, this very same information, when viewed in the Visualizations tab, looks much more interesting, as seen in the following screenshot. There will be more on creating dashboard panels and dashboards in Chapter 6, Panes of Glass:

The eval command is perhaps the most advanced and powerful command in SPL. It allows you to store the resulting value of the eval operation in a field. A myriad of functions available today can be used with eval. Let us try some of the simpler and more common ones.

The simplest type of eval command performs a simple calculation and stores it in the newly created field. For example, if you want to create the new_salary field, which adds together old_salary plus a field named raise, it would look like this (but don't try this, as there are no such fields in our data):

SPL> eval new_salary = old_salary + raise

There are also countless functions that can be used effectively with eval. Later we discuss some of them:

SPL> round(X, Y)

Run the search command below, then modify it to include the eval function round(X, Y). Watch how the percent column values were transformed as they are rounded to the nearest integer with two decimal values:

SPL> index=main | top url 
 
     index=main | top url | eval percent=round(percent, 2) 
 
     upper(X)

Use this function to transform the URL strings into uppercase:

SPL> index=main | top url 
 
     index=main | top url | eval url=upper(url) 
 
     case(X, "Y", ...)

The case function is especially useful when transforming data based on a Boolean condition. If X is true, then assign to the variable the string Y. Here is one example:

SPL> index=main | top url showperc=false  
     | eval Tag=case(url=="/home", "Home", url="/auth", "Auth")

The resulting table shows that a new column called Tag has been created and all instances of /home have been marked as Home and all instances of /auth have been marked as Auth:

The rex or regular expression command is extremely useful when you need to extract a field during search time that has not already been extracted automatically. The rex command even works in multi-line events. The following sample command will get all the versions of the Chrome browser that are defined in the highlighted User Agent string part of the following raw data. Let's say this is your raw data, and you need to get the highlighted value:

016-07-21 23:58:50:227303,96.32.0.0,GET,/destination/LAX/details,-,80, 
-,10.2.1.33,Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) 
AppleWebKit/537.36 (KHTML; like Gecko) Chrome/29.0.1547.76 
Safari/537.36,500,0,0,823,3053 

You can use this search command to get it:

SPL> index=main | rex field=http_user_agent 
     "Chrome/(?<Chrome_Version>.+?)?Safari" | top Chrome_Version

The rex command extracted a field called Chrome_Version during the search and made it available for all succeeding commands. The results are shown in the following screenshot:

In this chapter, we introduced you to the SPL. You have learned that the search pipeline is crucial in the transformation of data as it is piped between search commands and eventually to the final results table. We also introduced you to time modifiers and how to filter search results. Lastly, you were introduced to multiple search commands that are commonly used. In Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches, we will go on to use our search processing skills to create useful reports and learn about developing alerts that will increase organizational efficiency and prevent errors. We will also learn more about how to best optimize our searches.

When you enable acceleration for a data model, Splunk internally summarizes the data defined by the data model in a given time range. This gives a tremendous boost to the search speed for your data model. There are a couple of things to remember when you enable data model acceleration:

In this exercise, you will enable data model acceleration with a summary range of 7 days.

You will want to follow these steps very carefully:

Now that the data model has been fully constructed and accelerated, it is time to use it with the Pivot Editor.

The Pivot Editor

Now we will begin to make a Pivot table; follow these directions:

1. Go back to the Destinations app and click on Pivot in the main menu.
2. This time, simply click on the WebLogs object. You will see a page as shown in the following screenshot with a count of all WebLogs data for All Time:
   

We have highlighted different sections in this page. The navigation bar icons to the left of the screen represent the different visualization modes. The default and topmost visualization is the statistics table. You will always first construct your statistics table before you go to any of the other visualizations.

The time range functions the same throughout Splunk. Always change it to something within the scope of your acceleration summary range (7 days in this case). Filters will allow you to narrow down your dataset based on object attributes.

Split Rows and Split Columns will allow you to change the orientation of your data based on Time and Attribute. The following screenshot shows you what attributes will appear on the Split Columns dropdown:

Column Values on the other hand will allow you to select an Event or Attribute based on Time, as shown in the following screenshot:

This will all look confusing at first so it is best to walk through it by means of examples.

In the upper-right corner of the page, you will see the scope of the Pivot. This is usually the object that you clicked when you first entered the Pivot Editor. Through this dropdown, you can switch to other data models and other objects, as shown in the following screenshot. It is also a shortcut for checking the status of acceleration:

The Pivot Editor will always default to the Count of the object that was selected in the data model. That is why, in the results section, you see Count of WebLogs:

Creating a chart from a Pivot

Let us create a chart that will represent web traffic. WebLogs that have been generated by Eventgen are simulated data from a web application with varied status codes. Every line in the web log is a request from a web client (such as the browser or mobile device). In effect, the sum of all the requests and status codes equals the entire traffic of the web application.

To create a chart, do the following:

1. First, change the time range to Last 7 days.
2. Change Split Rows to _time and leave Periods as the default, as shown next. This is equivalent to using the timechart function in SPL without specifying a span. Opt out of Totals if you do not want the bottom row of your dataset to include the summation of the entire column. In this case, we do not need Totals, so No is selected:
   
3. Click Add To Table. The results will change to show seven rows, with the sum of the WebLogs tallied in each row. In Split Columns, add the http_status_code attribute to split the columns based on http_status_code. There will be many options available to you to tweak your data set for the Split Columns function, but for now leave them as they are. The final selection of filters is shown in the following screenshot:
   

Your statistics table should have changed. The count per day has now been split based on the http_status_code attribute, as shown next. Since your data model is accelerated and you have selected the time filter that fits the summary range, the results should render almost instantaneously:

Creating an area chart

Now that the statistics table has been populated with data, it is time to choose a visualization method:

1. Select the Area visualization tool in the left menu bar, as shown in the following screenshot:
   

   The next page will show you an array of options that you can choose from to change the way your area chart behaves. Depending on how long you have been running Splunk on your machine, you should see a stacked area chart similar to the following screenshot:

   
2. Let us change the look and feel of the area chart a little bit. In the X-Axis (Time) section, choose to hide the Label. This will remove the _time label on the x axis:
   
3. In the Color (Areas) section, let's move the Legend Position to the bottom, as shown in the following screenshot:
   
4. Your area chart is now ready to be saved as a Dashboard panel. Click on the Save As button and select Dashboard Panel.
5. Let us create a new dashboard called Summary Dashboard. Make sure you change the permission to Shared in App.
6. Finally, change the Panel Title to Web Traffic per Day Last 7 Days.
7. Click on Save to finish and click on View Dashboard. Use the following screenshot as a guide:
   

You now have a dashboard that is driven by the data model that you just created. It should look similar to what is shown here:

Before going on, you can rearrange your dashboard so that it looks the way you want:

By now, you have familiarized yourself with data models and the Pivot Editor. In this chapter, we explained what data models are and how they are created. We walked you through how to create your data model objects based on a hierarchy. You also have learned that data models consist of attributes that can be inherited from the parent objects. You created an attribute by extracting a field using regular expression. We have also shown you how to use the Pivot Editor and create three different visualizations: area chart, pie chart, and single value with trend sparkline.

In the next chapter, Chapter 5, Data Optimization, Reports, Alerts, and Accelerating Searches, you will learn how to create and use these important Splunk tools as well as how to optimize searches.

Tags in Splunk are useful for grouping events with related field values. Unlike event types, which are based on specified search commands, tags are created and mapped to specific fields. You can also have multiple tags assigned to the same field, and each tag can be assigned to that field for a specific reason.

The simplest use-case scenario when using tags is for classifying IP addresses. In our Eventgen logs, three IP addresses are automatically generated. We will create tags against these IP addresses that would allow us to classify them based on different conditions:

In our server farm of three servers, we are going to group them by purpose, patch status, and geolocation. We will achieve this using tags, as shown in the following steps:

This will give you all the events that come from the servers that are patched and hypothetically located in the east side of a building. You can then combine these with other search commands or an event type to narrow down the search results.

Consider a scenario where you need to find all booking payments with errors originating from the servers in the east side of a hypothetical building.

Without event types or tags, you would create a search command that looked something like this:

SPL> index=main server_ip=10.2.1.33 OR server_ip=10.2.1.35  
     AND (http_uri=/booking/payment http_status_code=500)

Compare that to this much more elegant and shorter search command, which you can try now:

SPL> index=main eventtype=bad_payment tag=east

Here's an additional exercise for you. Create tags for the following fields using this table as a guide and use them in a search query:

Now you can use these tags to search for bookings to major destinations, which have a status code of not_found. These tags can make your searches much easier and more useful. Here is an example of a search command that combines what you have learned in this chapter so far:

Occasionally you will come across pieces of data that you wish were rendered in a more readable manner. A common example is HTTP status codes. Computer engineers are often familiar with status codes as three-digit numbers. Business analysts, however, would not necessarily know the meaning of these codes. In Splunk, you solve this predicament by using lookup tables, which can pair numbers or acronyms with more understandable text classifiers.

A lookup table is a mapping of keys and values that Splunk can query so it can translate fields into more meaningful information at search time. This is best understood through an example. You can go through the following steps:

The new lookup table file path will now appear in the main Lookup Table Files page. Change the permission so that all apps can use it and it will now appear as Global. The entries of the lookup table files should be similar to the following screenshot:

Now that we have configured the lookup table file, it is time to define the lookup:

Let us now try to make use of this new lookup table:

This is good for a first step, but for it to be a practical tool, the lookup needs to happen automatically with all queries. To do this, take the following steps:

Now let's see how these changes can help us out:

So far in this chapter, you have learned how to do three very important things: classify data using event types, normalize data using tags, and enrich data using lookup tables. All these, in addition to Chapter 4, Data Models and Pivot, constitute the essential foundation you need to use Splunk in an efficient manner. Now it is time to put them all to good use.

Splunk reports are reusable searches that can be shared to others or saved as a dashboard. Reports can also be scheduled periodically to perform an action, for example to be sent out as an e-mail. Reports can also be configured to display search results in a statistical table, as well as visualization charts. You can create a report through the search command line or through a Pivot. Here we will create a report using the search command line:

You can modify the permissions so others can use your report. You have done this step a couple of times earlier in the book. This process will be identical to editing permissions for other objects in Splunk.

You can create a schedule to run this report on a timely basis and perform an action on it. The typical action would either be sending the result as an e-mail or running a script. Unfortunately, you would need a mail server to send an e-mail, so you will not be able to do this from your Splunk workstation the way it is currently configured. However, we will show you how it is done:

There is another option that you will commonly use for reports adding them to dashboards. You can do this with the Add to Dashboard button. We will use this option in Chapter 6, Panes of Glass.

Create a few more reports from SPL using the following guidelines. We will use some of these reports in future chapters so try your best to do all of them. You can always come back to this chapter if you need to:

You also have the ability to create reports using Pivot:

Alerts are crucial in IT operations. They provide real-time awareness of the state of the systems. Alerts also enable you to act fast when an issue has been detected prior to waiting for a user to report it. Sure enough, you can have a couple of data center operators monitor your dashboards, but nothing jolts their vigil more than an informative alert.

Now, alerts are only good if they are controlled and if they provide enough actionable information. Splunk allows you to do just that. In this section, we will walk you through how to create an actionable alert and how to throttle the alerting to avoid flooding your mailbox.

The exercises in this section will show you how to create an alert, but in order to generate the actual e-mail alert, you will need a mail server. This book will not cover mail servers but the process of creating the alert will be shown in full detail.

We want to know when there are instances of a failed booking scenario. This event type was constructed with the 500 HTTP status code. 5xx status codes are the most devastating errors in a web application so we want to be aware of them. We will now create an alert that will be triggered when a bad booking event is detected. Follow these steps:

Let us explain some of the different options in this selection:

Click on Save to save your first alert. We will come back later to optimize it. Meanwhile, you should have now been sent to the alert summary page where you can continue to make changes. Note that since we selected the Add to Triggered Alerts action, you should now see the history of when this alert was triggered on your machine. Since the Eventgen data is randomized and we scheduled it to run every hour, you may have to wait until the next hour for results:

In Chapter 4, Data Models and Pivot, you learned how to accelerate a data model to speed up retrieval of data. The same principle applies to saved searches or reports:

No matter how advanced and well-scaled your Splunk infrastructure is, if all scheduled searches and reports are running at the same time, the system will start experiencing issues. Typically you will receive a Splunk message saying that you have reached the limit of concurrent or historical searches. Suffice to say that there are only a certain number of searches that can be run on CPU core for each Splunk instance. The very first issue a beginner Splunk admin faces is how to limit the number of concurrent searches running at the same time. One way to fix this is to throw more servers into the Splunk cluster, but that is not the efficient way.

The trick to establishing a robust system is to properly stagger and budget scheduled searches and reports. This means ensuring that they are not running at the same time. There are two ways to achieve this:

Let us see an example of how to use a custom Cron schedule. Begin with this search query, which finds all errors in a payment:

The Cron expression * * * * * corresponds to minute hour day month day-of-week.

Learning Cron expressions is easiest when you look at examples. The more examples, the simpler it is to understand this method of scheduling. Here are some typical examples:

Now that you know something about Cron expressions, you can fine-tune all your searches to run in precise and different schedules.

In a matter of days, Splunk will accumulate data and start to move events into the cold bucket. If you recall, the cold bucket is where data is stored to disk. You will still be able to access this data but you are bound by the speed of the disk. Compound that with the millions of events that are typical with an enterprise Splunk implementation, and you can understand how your historical searches can slow down at an exponential rate.

There are two ways to circumvent this problem, one of which you have already performed: search acceleration and summary indexing.

With summary indexing, you run a scheduled search and output the results into an index called summary. The result will only show the computed statistics of the search. This results in a very small subset of data that will seemingly be faster to retrieve than going through the entirety of the events in the cold bucket.

Say, for example, you wish to keep track of all counts of an error in payment and you wish to keep the data in the summary index. Follow these steps:

Now perform the following steps:

Use the following screenshot as a guide:

Now perform the following steps:

Use the following information as a guide:

      SPL> index=summary search_name="Summary of Payment Errors"

Notice that this stripped the original event of all other information except the count of events at the time that the scheduled search is running. We will use this information in later chapters to create optimized dashboards.

In this chapter, you have learned how to optimize data in three ways: classifying your data using event types, normalizing your data using tags, and enriching your data using lookup tables. You have also learned how to create advanced reports and alerts. You have accelerated your searches just like you did with data models. You have been introduced to the powerful Cron expression, which allows you to create granularity on your scheduled searches, and you have also been shown how to stagger your searches using time windows. Finally, you have created a summary index that allows you to search historical data faster. In the next chapter, Chapter 6, Panes of Glass, you will go on to learn more about how to do visualizations.

There are three kinds of dashboard that you will typically create with Splunk:

Dynamic form-based dashboards allow Splunk users to change the dashboard data without leaving the page. This is accomplished by adding input fields (such as time, radio (button), textbox, checkbox, dropdown, and so on) in the dashboard, which changes the data based on the current selection. This is an effective type of dashboard for teams that troubleshoot issues and analyze data.

Static real-time dashboards are often kept on a big panel screen for constant viewing, simply because they are so useful. You see these dashboards in data centers or Network Operations Centers (NOCs). Even though they are called static, in fact the data changes in real time without refreshing the page; it is just the format that stays constant. The dashboard will also have indicators and alerts that allow operators to easily identify a problem and act on it. Dashboards like this usually show the current state of the network or business systems, using indicators for web performance and traffic, revenue flow, and other important measures.

Dashboards as scheduled reports are the only kind that breaks away from the rules mentioned previously. This type will typically have multiple panels included on the same page. Also, the dashboard will not be exposed for viewing; it will generally be saved as a PDF file and sent to e-mail recipients at scheduled times. This format is ideal when you need to send information updates to multiple recipients at regular intervals.

In this chapter, we will create these three types of dashboard. You will also learn how to use and interact with the Splunk Dashboard Editor to develop advanced visualizations.

Dynamic form-based dashboard

In this section, we will create a dynamic form-based dashboard in our Destinations app that will allow users to interact with form inputs to change and redisplay the data. Here is a screenshot of the final output of this dynamic form-based dashboard:

Dynamic dashboard with form input

Let's begin by creating the dashboard itself and then generating the base panels:

1. Open the Destinations app.
2. Run this search command:

   SPL> index=main status_type="*" http_uri="*" server_ip="*" 
          | top status_type, status_description, http_uri, server_ip
   

   Note

   Important

   Be careful when copying commands with quotation marks. It is best to type in the entire search command to avoid problems.

3. Click on Save As | Dashboard Panel.
4. Fill in the information based on the following screenshot:
   
5. Click on Save.
6. Close the pop-up window that appears.

Creating a Status Distribution panel

We will go to the dashboard later, once all our panel searches have been generated. Let us go ahead and create the second panel:

1. In the search window, type in the following search command:

   SPL> index=main status_type="*" http_uri=* server_ip=* 
              | top status_type
   

2. You will save this as a dashboard panel to the newly-created dashboard. In the Dashboard option, click on the Existing button and look for the new dashboard, as seen here. Don't forget to fill in the Panel Title as Status Distribution:
   
3. Click on Save when you are done.

Creating the Status Types Over Time panel

Now we'll move on to create the next panel:

1. Type in the following search command:

   SPL> index=main status_type="*" http_uri=* server_ip=* 
              | timechart count by http_status_code
   

2. You will save this as a Dynamic Form-based Dashboard panel as well. Type in Status Types Over Time in the Panel Title field:
   
3. Click on Save.

Creating the Hits vs Response Time panel

Now on to the next panel. Use the following search command:

SPL> index=main status_type="*" http_uri=* server_ip=* 
| timechart count, avg(http_response_time) as response_time

Save this dashboard panel as Hits vs Response Time:

Arranging the dashboard

Now, we'll go on to look at the dashboard we've created and make a few changes.

1. Click on the View Dashboard button. If you missed out on the View Dashboard button, you can find your dashboard by clicking on Dashboards in the main navigation bar.
2. Let us edit the panel arrangement. Click on Edit | Edit Panels.
3. Move the Status Distribution panel to the upper-right row.
4. Move the Hits vs Response Time panel to the lower-right row.
5. Click on Done to save your layout changes.

Look at the following screenshot. The dashboard framework you've created should now look much like this.

The dashboard probably looks a little plainer than you expected it to. But don't worry about how it looks for now. We will fix the dashboard one panel at a time.

Dynamic dashboard with four panels in tabular formats

Now that we have the layout framework in place, let us start modifying the panels. The first panel is how we want it to look so we do not need to change it.

Panel options

In this section, we will learn how to alter the look of our panels and create visualizations in them.

Go to Edit mode by clicking on Edit | Edit Panels.

Each dashboard panel will have three setting options to work with: inline search options, visualization type, and visualization options. They are represented by three dropdown icons.

The INLINE SEARCH drop-down allows you to modify the title, change the search string, change the time modifier for the search string, convert the panel into a report, and delete the panel.

The Visualization Type drop-down allows you to change the type of visualization to use for the panel, as shown in the following screenshot:

Finally, the Visualization Options drop-down will give you the ability to fine-tune your visualization. These options will change depending on the visualization you select. For a normal statistics table, this is how it will look.

Pie chart - status distribution

Go ahead and change the Status Distribution visualization panel to a pie chart. You do this by selecting the Visualization Type dropdown and selecting Pie. Once done, the panel will look like the following screenshot:

Stacked area chart - Status Types Over Time

Change the Status Types Over Time panel to Area. By default, area charts will not be stacked. Let us fix this by clicking on the Visualization Options dropdown.

1. In the Stack Mode section, click on Stacked. For Null Values, select Zero. Use the chart that follows for guidance.
   
2. Click on Apply. The panel will change right away.
3. Let us clean it up further. Let us remove the _time label as it is already implied. You can do this in the X-Axis section by setting the Title to None.
   

Here now is the new stacked area chart panel.

Now that we have the dashboard layout that we want, it is time to make it dynamic and interactive. Before we proceed, let us just highlight some of the basic key concepts related to form inputs.

Just as in any web page, a form input is an element that allows you to select or type in information that will be submitted to the application for processing. There are different form inputs available for Splunk dashboards:

If you click on Edit | Edit Panels, you will see that you can select which Form Input you require by clicking on the Add Input dropdown.

Although the options are generally different for every type of input, there are common concepts that you need to fully comprehend. So it is worth looking at this list carefully before we take you through some examples.

In the General section, you'll see the following options:

In the Token Options section, you'll see the following option:

In the Static Options section, you'll see the following option:

In the Dynamic Options section, you'll see the following options:

Let us change our input field into a time range field.

You can now try to change the time range using the time input, but nothing will happen. This is because we have not yet configured the panels to react when the time input has been changed. Let us do that now:

Notice that the data on the first panel now reacts to the changes you make on the time range input. Perform the same steps on the other three panels and watch the data change based on your selected time range.

Now we are going to create radio inputs with dynamic search options. This will allow viewers to select server and status types, and will affect the information rendered by the panels:

Now that you have configured the radio input with dynamic search options, you will see that the selection has been populated, along with the static option that you created. This is a great way of creating selection inputs when you know that the items will regularly change depending on a search query:

Try this exercise to reinforce what you have learned. Create a second radio input option, following the same steps as previously, with the following information:

Click on Apply  to save your changes.

If you did it correctly, the newly-created radio input will look like this:

Similar to when we first created the Time input, the panels will not react to these new inputs until we associate them. In order for the new inputs to change the values in the panels, we have to modify each panel search string to include the new tokens serverand status_type:

Here is an example of data being filtered to 10.2.1.34 and Redirection for data arriving in the last 60 minutes:

At this point, you will appreciate what form input does to your dashboard. By simply substituting tokens in your search string, you are dynamically altering the panel charts so your users can filter the data in the ways they need. Continue editing the remaining panels using the following guide. You can refresh your browser if the changes do not happen right away.

Dropdown inputs function exactly the same as radio inputs. The former is used when the selection is huge and you do not want the list of choices to unnecessarily clutter the entire page. The http_uri field has numerous results, so this makes a drop-down the ideal candidate for input here.

Follow the same procedure as for radio input creation, but make sure you have selected Dropdown instead. Use the following information and screenshots as guides to complete the task:

If done correctly, the newly-created drop-down input will look like this:

Now that you have created the inputs, go ahead and associate them with the search panels. The same procedure applies; you have to edit each search string to include the new token:

When all the form inputs are done, this is how it should look. We show the heading, where you can filter, first:

This is now a fully-functional, dynamic, form-based dashboard. It was a lot of work, but the more you do it, the easier the process becomes.

Dynamic form-based dashboard with four chart panels

In this section, we will create a real-time dashboard that will display crucial information based on the data we have. To encourage you, we present a screenshot here with how it will look when we are done:

Test real-time dashboard with advanced indicators, combo charts, and choropleth charts

Single Value Panels with trends

We will now create two more single value panels that indicate trend lines. This is useful when you need your viewer to understand the behavior of the data in a very compressed line chart while highlighting the most current value. This is commonly used in viewing financial stock prices:

1. Enter edit mode with Edit | Edit Panels.
2. Create a clone of the Bookings panel. Follow the steps in the previous section.
3. Add it to the dashboard.
4. Rename the new panel Errors.
5. Change the Search String to the following command:

         SPL> index=main http_status_code=5* | timechart count
   

6. Click on the Visualization Options dropdown.
7. Set Under Label to last 60 mins.
8. Click on Yesfor Show Trend Indicator.
9. In the Compared to dropdown, select 1 hour before.
10. Ensure that Show Sparkline is set to Yes.
11. Refer to the following screenshot:
    
12. Click on Apply.
13. Click on the Search dropdown | Edit Search String.
14. In Select Time Range, click Real-time.
15. Change the Earliest value to 24 Hours Ago:
    
16. Click on Apply, then Save.
17. Drag the panel to the right end of the first row.
18. After following the previous steps, click on the Done button.

Repeat the previous procedure to create another panel. Use the following information to build the new panel:

• Title: Response Time
• Search String: index=main | timechart avg(http_response_time) as response_time span=1h
• Time Range: Real-time 24 Hours Ago
• After Label: ms
• Under Label: compared to an hour ago
• Show Trend in: Percent

The new single value panels have been created and are packed with information. First you see the current value within the last hour, then you see an upward or downward trend, and finally you see a sparkline (or trend line) that spans 24 hours.

The first row will now look similar to the following screenshot: