Splunk 7 Essentials - Third Edition

5 (3 reviews total)
By J-P Contreras , Erickson Delgado , Betsy Page Sigman
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Splunk – Getting Started

About this book

Splunk is a search, reporting, and analytics software platform for machine data, which has an ever-growing market adoption rate. More organizations than ever are adopting Splunk to make informed decisions in areas such as IT operations, information security, and the Internet of Things.

The first two chapters of the book will get you started with a simple Splunk installation and set up of a sample machine data generator, called Eventgen.

After this, you will learn to create various reports, dashboards, and alerts. You will also explore Splunk's Pivot functionality to model data for business users. You will then have the opportunity to test-drive Splunk's powerful HTTP Event Collector. After covering the core Splunk functionality, you'll be provided with some real-world best practices for using Splunk, and information on how to build upon what you've learned in this book.

Throughout the book, there will be additional comments and best practice recommendations from a member of the SplunkTrust Community, called "Tips from the Fez".

Publication date:
March 2018


Splunk – Getting Started

Splunk is a multinational software company that offers its core platform, Splunk Enterprise, as well as many related offerings built on the Splunk platform. Cofounded by Michael Baum, Rob Das, and Erik Swan, Splunk's name was inspired by the process of exploring caves, or spelunking. The Splunk platform helps a wide variety of organizational personas, such as analysts, operators, developers, testers, managers, and executives. They get analytical insights from machine-created data. Splunk collects, stores, and provides powerful analytical capabilities, enabling organizations to act on often powerful insights derived from this data.

The Splunk Enterprise platform was built with IT operations in mind. When companies had IT infrastructure problems, troubleshooting and solving problems was immensely difficult, complicated, and manual. Splunk was built to collect and make log files from IT systems searchable and accessible. Splunk is commonly used for information security and development operations, as well as more advanced use cases for custom machines, Internet of Things, and mobile devices.

Throughout the book, we will be covering the fundamental concepts of Splunk so that you can learn quickly and efficiently. As the concepts become more complex, we reserve their deep discussion for Splunk's online documentation or the vibrant Splunk online community at http://docs.splunk.com. Wherever necessary, we provide links to help provide you with the practical skills and examples so that you can get started quickly.

With very little time, you can achieve direct results using Splunk, which you can access through a free enterprise trial license. While this license limits you to 500 MB of data ingested per day, it will allow you to quickly get up to speed with Splunk and learn the essentials of this powerful software.

If you follow what we've written closely, we believe you will quickly learn the fundamentals you need to use Splunk effectively. Together, we will make the most of the trial license and give you a visible result that you can use to create valuable insights for your company.

Towards the end of the book, we will cover concepts to extend Splunk to your organization, and cover the vast Splunk community and online ecosystem.

Tip from the Fez: Splunk sponsors a community known as Splunk Trust. Splunk Trust is made up of bright Splunk minds from around the world, who actively and successfully participate in the Splunk community, especially through the Splunk answers online site. A logo associated with this community is based on the historical headdress and named after the city of Fez, Morocco. Many fraternal organizations have adopted the use of a Fez, most notably in the United States by the Shriners organization. Throughout this book, we will tap into one of Splunk Trust's members to provide some additional best practice recommendations.

Your Splunk account

First, you will need to register for a Splunk account; go to https://www.splunk.com/. This is the account that you will use if you decide to purchase a license later. Go ahead and do this now. From here on, the password you use for your Splunk account will be referred to as your Splunk password.

Obtaining a Splunk account

To obtain your Splunk account, perform the following steps:

  1. Go to the Splunk sign up page at http://www.splunk.com.
  2. In the upper-right corner, click on the Free Splunk button.
  3. Enter the information requested.
  4. Create a username and password.
  5. You will be taken to the Splunk download page, where you will need to click on the Free Download button under Splunk Enterprise as shown in the following screenshot:
  1. You will then need to download the Splunk Enterprise software. Go to http://download.splunk.com and select the Splunk Enterprise free download. Choose your operating system, being careful to select 32 or 64-bit (whichever is appropriate in your case; most should select 64-bit, which most computers of today use). For Windows, download the *.msi file. For Linux, download the *.tgz file. In this book, we work with version 7.0.1.

The installation is very straightforward. Follow the steps for your particular operating system, whether it is Windows or Linux.

This book assumes a single Splunk standalone installation. Make sure that there is no previous installation of Splunk in your system. If there is any, uninstall the prior version before proceeding with the next steps.

Installing Splunk on Windows

These are the instructions you need to follow to install Splunk on your Windows desktop. Take your time and do not rush the installation. Many chapters in this book will rely on these steps:

  1. Run the installer that you downloaded.
  2. Check the box to accept the License Agreement and then click on Customize Options, as shown in the following screenshot:
  1. Change the installation path to C:Splunk. You will thank us later as it simplifies issuing Splunk CLI (command line interface) commands. This is also a best practice used by modern Windows administrators. Remember to eliminate white spaces in directory names as well, as it causes complications with scripting. Click on Next to continue, as seen in this screenshot:
  1. Install Splunk Enterprise as Local System and then click on Next.
  2. Leave the checkbox selected to Create Start Menu Shortcut.
  3. Click on Install.
  4. Wait for the installation to complete.
  5. Click on Finish to complete the installation. It will attempt to launch Splunk for the first time in your default browser.
Throughout the book, you will see references to $SPLUNK_HOME. This will be the installation directory of Splunk. In Windows, as a convention used in this book, $SPLUNK_HOME will be at C:Splunk.
For the Linux convention used in this book, $SPLUNK_HOME is where Splunk is installed in the particular user's home directory. Large-scale Splunk deployments should and will deviate from these personalized settings.

Installing Splunk on Linux

If you choose to install Splunk on a Linux machine, these are the instructions you need to follow. Take your time and do not rush the installation. For this Linux installation, the steps assume you will run Splunk in your user profile's home directory:

  1. Decompress the .tgz file you downloaded. The result of the decompression is a splunk folder and all the related Splunk files contained inside.
  2. Change the working directory to $SPLUNK_HOME/bin.
  3. Run the following command to start Splunk for the first time:
./splunk start 
  1. Accept the Splunk license when prompted.
  2. Splunk will start successfully and the end results should appear like this:
Tip from the Fez: In most organizational environments, it is common to install Splunk in /opt/. When installing on Linux, it is best practice to adjust the ulimit settings and disable transparent hugh page functionality for Splunk applications that will be loading data in real time, supporting a community of users.
Tip from the Fez: Unlike Windows installations, Splunk Linux installations aren't automatically configured to start upon reboot of the Splunk server. Ensure that you configure your Splunk instance on Linux for what is known as boot start so that if the Linux instance is restarted, Splunk will automatically restart with the operating system.
Tip from the Fez: In an enterprise environment, create a Linux user specifically for running Splunk and use that. Do not run Splunk as the Linux root user or an individuals user.

Logging in for the first time

Launch the application for the first time in your default browser. You can also manually access the Splunk web page via the http://localhost:8000 URL if you have installed locally. If using a cloud instance, use http://ipaddress:8000 since you are connecting via a traditional internet connection as opposed to locally.

Splunk requires you to use a modern browser. It supports most versions of Google Chrome, Firefox, and newer versions of Internet Explorer. It may not support older versions of Internet Explorer.

Log in with the default username as admin and password as changeme, as indicated in the following screenshot:

The next step is to change the default administrator password, while keeping the default username. Do not skip this step. Make security an integral part of your day-to-day routine. Choose a password that will be secure.

Assuming that all goes well, you will now see the default Splunk Enterprise landing page as follows:

Running a simple search

You are ready to run your first Splunk search:

  1. Click directly on the green Search & Reporting app button. In this example, you use Splunk's very own internal index; this is Splunk's way of splunking itself (or collecting detailed information on all its underlying processes).
An index is the term given to where Splunk stores event data captured from log files, APIs, HTTP events, delimited files, and other machine data sources. It is in some ways like a database, but should not be compared to traditional relational databases in functionality or performance.
  1. In the New Search input, type in the following search query (more about the Search Processing Language (SPL) in Chapter 3, Search Processing Language):
SPL> index=_internal sourcetype=splunkd
The SPL> prefix will be used as a convention in this book to indicate a Splunk Search command. Since SPL is submitted via the Splunk user interface, there is no difference whether operating in Windows or Linux environments. The underscore before the index name _internal means that it is a system index internally used by Splunk. Omitting the underscore will not yield any result, as internal is not a default index.
  1. This search query will have as an output the raw events from the metrics.log file that is stored in the _internal index. A log file keeps track of every event that takes place in the system. The _internal index keeps track of every event that occurs and makes it easily accessible.
  1. Take a look at these raw events, as shown in the following screenshot. You will see fields listed on the left side of the screen. The important Selected Fields are host, source, and sourcetype. We will go into more detail about these later, but suffice to say that you will frequently search on one of these, as we have done here. As you can see from the highlighted fields, we indicated that we were looking for events where sourcetype=splunkd. Underneath Selected Fields, you will see Interesting Fields. As you can tell, the purposes of many of these fields are easy to guess as seen in the following screenshot:

Creating a Splunk app

It is good practice to create a custom Splunk app to isolate the configurations you add to Splunk. You may never have created an app before, but you will quickly see it is not very difficult. Here, we will create a basic app called Destinations that we will use throughout this book:

  1. Let's access the Manage Apps page. There are two ways to do this; you can either click on the Apps icon at the home page as shown in the following screenshot:
  1. Or, select Manage Apps from the app dropdown in the top navigation bar of the Search & Reporting app:
  1. At the Manage Apps page, click on the Create app icon shown in the following screenshot:
  1. Finally, populate the forms with the following information to complete app creation. When you are done, click on the Save button to create your first Splunk app:
  1. You have just created your very first Splunk app. Notice that it now appears in the list of apps and it has a status of Enabled, meaning it is ready to be used:

We will use this app to complete the exercises in this book, but first we need to make a few important changes:

  1. Click on the Permissions link shown in the preceding screenshot
  2. In the next window, under the Sharing for config file-only objects section, select All apps

These steps will ensure that the application will be accessible to the Eventgen add-on, which will be installed later in the chapter. Use the following screenshot as a guide:

Splunk permissions are always composed of three columns: Roles, Read, and Write. A role refers to certain authorizations or permissions that can be taken on by a user. Selecting Read for a particular role grants the set of users in the role permission to view the object. Selecting Write will allow the set of users to modify the object. In the preceding screenshot, everyone (all users) will have access to view the Destinations app, but only the admin (you) and a power user can modify it.


Populating data with Eventgen

Machine data is produced by the many functions carried out by computers and other mechanical machines. If you work in an environment that is rich in machine data, you will most likely have many sources of readily available data inputs for Splunk. However, to facilitate learning in this book, we will use a Splunk add-on called the Splunk Eventgen to easily build real-time and randomized web log data. This is the type of data that would be produced by a web-based e-commerce company.

If you need more detailed information about Eventgen, you can follow the project's GitHub repository at https://github.com/splunk/eventgen/.

Using the CLI to configure Eventgen

Here's an important tip for Windows users. Make it a habit to always launch your Command Prompt in administrator mode. This allows you to use commands that are unhindered by Windows security:

  1. Right-click on the Windows start menu icon and select Search. In Windows 7, you can click on the Windows icon and the search window will be directly above it. In Windows 10, there is a search bar named Cortana next to the Windows icon that you can type into. They both have the same underlying function.
  2. In the search bar, type cmd.
  3. In the search results, look for command.exe (Windows 7) or Command Prompt (Windows 10), right-click on it, and then select Run as administrator.
Familiarize yourself with this step. Throughout the rest of the book, you will be frequently asked to open Command Prompt in administrator mode. You will know if you are in administrator mode, as it will say administrator: Command Prompt in the title of the Command Prompt window.

Installing the Eventgen add-on (Windows and Linux)

A Splunk add-on extends and enhances the base functionality of Splunk. An add-on typically enriches data using prebuilt rules for a specific source to enable faster analysis. In this section, you will be installing your first add-on, called Splunk Eventgen, which will help us pre-populate Splunk with real-time simulated web data:

  1. Download the ZIP file from the Eventgen public repository, http://github.com/splunk/eventgen. Click on the green Clone or download button. For Linux users, we've placed a copy of the Eventgen files used in this book at our download site: https://github.com/PacktPublishing/Splunk-7-Essentials-Third-Edition.
  2. Extract the ZIP file to the root location for your environment.
  3. Rename the extracted folder to SA-Eventgen. In Windows, this can be done with the Windows GUI, or via the mv command in Linux.
  4. Open an administrator Command Prompt or Linux shell and execute the following command (the slashes are important):
Windows: C:> xcopy SA-Eventgen C:\Splunk\etc\apps\SA-Eventgen /O /X /E /H /K
mv SA-Eventgen /$SPLUNK_HOME/etc/apps/
  1. In the prompt, type the following directory command to verify that the copy works properly and the contents are in the folder:
Windows: C:> dir C:\Splunk\etc\apps\SA-Eventgen
ls -l /$SPLUNK_HOME/etc/apps/

These are the contents of the recently copied SA-Eventgen folder, as shown in the following (Windows) screenshot, the same as what would appear in the similar location on Linux:

  1. Return to the Splunk interface in your web browser and restart Splunk by selecting the Settings dropdown; under the SYSTEM section, click on Server controls:

  1. On the Server controls page, click on the Restart Splunk button. Click on OK when asked to confirm the restart.
  2. The web interface will first notify you that Splunk is restarting in the background, then it will tell you that the restart has been successful. Every time Splunk is restarted, you will be prompted to log in with your credentials. Go ahead and log in.
  3. After logging in, you will now see the Eventgen app in the Splunk Enterprise landing page. Go to the Manage Apps page and confirm that the SA-EventGen application is installed:

You have successfully installed a Splunk add-on.


Controlling Splunk

There are several different ways to stop, start, or restart Splunk. The easiest way to restart Splunk is to do it from the web interface, as demonstrated in the preceding section. The web interface, however, only allows you to restart your Splunk instance. It does not offer any other control options.

The most flexible way to control Splunk is by using the CLI. Using the CLI is an essential skill for Splunk administrators.

In the console or Command Prompt, type in the following command and hit Enter on your keyboard:

Windows: C:> cd C:Splunk\bin
Linux: cd /$SPLUNK_HOME/bin

While in the $SPLUNK_HOME/bin directory, issue the following command to restart Splunk:

Windows: C:Splunk\bin>splunk restart
Linux: [[email protected] bin]$ ./splunk restart

After issuing this command, splunkd will go through its restart process. Here are the other basic parameters that you can pass to the Splunk application to control Splunk:


  • splunk status: Tells you whether splunkd is running or not
  • splunk stop: Stops splunkd and all its processes
  • splunk start: Starts splunkd and all its processes
  • splunk restart: Restarts splunkd and all its processes


  • ./splunk status: Tells you whether splunkd is running or not
  • ./splunk stop: Stops splunkd and all its processes
  • ./splunk start: Starts splunkd and all its processes
  • ./splunk restart: Restarts splunkd and all its processes

Doing this from a CLI gives the added benefit of verbose messages. A verbose message is a message with a lot of information in it. Such messages can be useful for making sure the system is working correctly or troubleshooting any errors.

A successful restart of splunkd generally has the following output (elements of which may vary):

In Windows, you can also control Splunk through the Splunkd Service, as shown in the following screenshot. The d in the service name, denoting daemon, means a background process. Note that the second service, splunkweb, is not running. Do not try to start splunkweb as it is deprecated and is there only for legacy purposes. The Splunk-running web application is now bundled in Splunkd Service:


Configuring Eventgen

We are almost there. Proceed by first downloading the exercise materials that will be used in this book. Open an administrator Command Prompt and make sure you are in the root of the Windows machine or Linux user shell. Download the ZIP file and extract it in your computer using https://github.com/PacktPublishing/Splunk-7-Essentials-Third-Edition.

The Eventgen configuration you will need for the exercises in this book has been packaged and is ready to go. We are not going into the details of how to configure Eventgen. If you are interested in learning more about Eventgen, visit the project page at http://github.com/splunk/eventgen.

Follow these instructions to proceed:

  1. Extract the project ZIP file into your local machine. Open an administrator console and use the change directory command to set where you extracted the file.
  2. Create a new samples directory in the Destinations Splunk app. The path of this new directory will be $SPLUNK_HOME/etc/apps/destinations/samples:
Windows: C:> mkdir C:\Splunk\etc\apps\destinations\samples
mkdir /splunk/etc/apps/destinations/samples
  1. Copy all the *.sample files from /labs/chapter01/eventgen of the extracted project directory into the newly created samples directory. Windows users can also copy and paste using a GUI:
Windows: C:> copy C:\splunk-essentials-         master\labs\chapter01\eventgen*.sample 
Linux: cp /splunk-essentials-
master/labs/chapter01/eventgen/*.sample /splunk/etc/apps/destinations/samples
  1. Now, copy the eventgen.conf into the $SPLUNK_HOME/etc/apps/destinations/local directory. Windows users can also copy and paste using the GUI if you prefer:
Windows: C:> copy C:\splunk-essentials-master\labs\     
chapter01\eventgen\eventgen.conf C:\Splunk\etc\apps\destinations\local
cp /splunk-essentials-master/labs/chapter01/eventgen.conf /splunk/etc/apps/destinations/local
  1. Grant the SYSTEM Windows account full access permissions to the eventgen.conf file. This is a very important step. You can either do it using the following icacls command or change it using the Windows GUI. This step (Step 5) is not required for Linux users, who can move on to Step 6:
C:> icacls C:\Splunk\etc\apps\destinations\local\eventgen.conf 
           /grant SYSTEM:F

A successful output of this command will look like this:

processed file: C:\Splunk\etc\apps\destinations\local\eventgen.conf
      Successfully processed 1 files; Failed processing 0 files
  1. Restart Splunk.

Viewing the Destinations app

Next we will see our Destinations app in action! Remember that we have configured it to draw events from a prototype web company. That is what we did when we set it up to work with Eventgen. Now, let's look at some of our data:

  1. After a successful restart, log back in to Splunk and proceed to your new Destinations app:
  1. In the Search field, type this search query and select Enter:
SPL> index=main

Examine the event data that your new app is enabling to come into Splunk. You will see a lot of references to browsers, systems, and so forth, the kinds of information that make a web-based e-commerce company run.

Try changing the time range to Real-time (5 minute window) to see the data flow in before your eyes:

Congratulations! You now have real-time web log data that we can use in subsequent chapters.

Tip from the Fez: Running a Splunk report under a real-time window places heavier strain on Splunk because it is rerunning the search over and over to generate the live nature of the real-time window. Unless absolutely needed, choose to have reports run for a set time period on user demand or a previously assigned schedule.

Creating your first dashboard

Now that we have data in Splunk, it's time to use it in order to derive something meaningful out of it. You are still in the Destinations app, correct? We will show you the basic routine when creating new dashboards and dashboard panels.

Type in or copy/paste the following search query in the Search Field, then hit Enter:

SPL> index=main /booking/confirmation [email protected] | timechart 
     count span=15m

After the search results render, click on the Visualization tab. This will switch your view into a visualization so you can readily see how your data will look. By default, it should already be using the Column Chart as shown in the following screenshot. If it does not, then use the screenshot as a guide on how to set it:

Now that you can see your Column Chart, it is time to save it as a dashboard. Click on Save As in the upper-right corner of the page, then select Dashboard Panel, as shown in the following screenshot:

Now, let's fill up that dashboard panel information, as seen in the following screenshot. Make sure to select the Shared in App in the Dashboard Permissions section:

Finish up by clicking View Dashboard in the next prompt:

You have created your very first Splunk dashboard with a single panel telling you the number of confirmed bookings in the last 24 hours at 15-minute intervals. If you let Eventgen run for a while and rerun the dashboard, the contents will populate as the data is generated. Time to show it to your boss!

In one chapter we acquired, installed, and used Splunk. We now have a fully-functional Splunk installation with live data. Leave Splunk running for two hours or so. After a few hours, you can stop Splunk if you need to rest for a bit to suppress indexing and restart it when you're ready to proceed into the next chapters.

Do you recall how to control Splunk from the command line?

Windows:     C:Splunkbin> splunk stop
             C:Splunkbin> splunk start
C:Splunkbin> splunk restart
Linux: bin]$ ./splunk stop
bin]$ ./splunk start
bin]$ ./splunk restart


In this chapter, you learned a number of basic Splunk concepts that you need to get started with this powerful tool. You learned how to install Splunk and configure a new Splunk app. You ran a simple search against the Splunk internal logs to ensure that the application is functional. You then installed a Splunk add-on called Eventgen, which you used to populate dummy data into Splunk in real time. You were shown how to control Splunk using the web user interface and the CLI. Finally, you created your very first Splunk dashboard, based off a single search. We will get a lot more creative with dashboards in additional chapters later in this book.

Now, we will go on to Chapter 2, Bringing in Data, to learn more about how to input data.

About the Authors

  • J-P Contreras

    J-P Contreras, a Splunk-certified administrator and sales engineer, has delivered value-oriented data analytics and performance planning solutions for 20+ years. He has built award-winning consulting teams to help companies turn data into analytical insights. He helps companies implement Splunk and enjoys everything the Splunk community offers.

    He received his MBA in e-commerce from DePaul University's Kellstadt Graduate School of Business, Chicago, in 2001. He trains in DePaul's Continuing Education Program and is a member of DePaul's Driehaus School of Business Advisory Board.

    He’d like to thank his family, especially his wife and children, and close friends for making life so enjoyable.

    Browse publications by this author
  • Erickson Delgado

    Erickson Delgado is an enterprise architect who loves to mine and analyze data. He began using Splunk in version 4.0 and has pioneered its use into his current work. He has worked with start-up companies in the Philippines to help build their open source infrastructure.

    He has developed applications with Python and node.js and is interested in Go and recovering programming with C/C++. In the recent years, he engaged himself in employing DevOps in his work.

    He blows off steam by saltwater fishing, mountain biking, crafting robots, and touring the country. He lives in Orlando.

    Browse publications by this author
  • Betsy Page Sigman

    Betsy Page Sigman is a distinguished professor at the McDonough School of Business at Georgetown University in Washington, D.C. She has taught courses in statistics, project management, databases, and electronic commerce for the last 16 years, and has been recognized with awards for teaching and service. She has also worked at George Mason University in the past. Her recent publications include a Harvard Business case study and a Harvard Business review article. Additionally, she is a frequent media commentator on technological issues and big data.

    Browse publications by this author

Latest Reviews

(3 reviews total)
Auch wenn es nicht die aktuellste Version ist, so werden die notwendigen Informationen beschrieben
Purchase and download of the book went off without a hitch.
Great overview of Splunk from the Analysts perspective.

Recommended For You

Splunk 7 Essentials - Third Edition
Unlock this book and the full library for FREE
Start free trial