Splunk is a multinational software company that offers its core platform, Splunk Enterprise, as well as many related offerings built on the Splunk platform. Cofounded by Michael Baum, Rob Das, and Erik Swan, Splunk's name was inspired by the process of exploring caves, or spelunking. The Splunk platform helps a wide variety of organizational personas, such as analysts, operators, developers, testers, managers, and executives. They get analytical insights from machine-created data. Splunk collects, stores, and provides powerful analytical capabilities, enabling organizations to act on often powerful insights derived from this data.
The Splunk Enterprise platform was built with IT operations in mind. When companies had IT infrastructure problems, troubleshooting and solving problems was immensely difficult, complicated, and manual. Splunk was built to collect and make log files from IT systems searchable and accessible. Splunk is commonly used for information security and development operations, as well as more advanced use cases for custom machines, Internet of Things, and mobile devices.
Throughout the book, we will be covering the fundamental concepts of Splunk so that you can learn quickly and efficiently. As the concepts become more complex, we reserve their deep discussion for Splunk's online documentation or the vibrant Splunk online community at http://docs.splunk.com. Wherever necessary, we provide links to help provide you with the practical skills and examples so that you can get started quickly.
With very little time, you can achieve direct results using Splunk, which you can access through a free enterprise trial license. While this license limits you to 500 MB of data ingested per day, it will allow you to quickly get up to speed with Splunk and learn the essentials of this powerful software.
If you follow what we've written closely, we believe you will quickly learn the fundamentals you need to use Splunk effectively. Together, we will make the most of the trial license and give you a visible result that you can use to create valuable insights for your company.
Tip from the Fez: Splunk sponsors a community known as Splunk Trust. Splunk Trust is made up of bright Splunk minds from around the world, who actively and successfully participate in the Splunk community, especially through the Splunk answers online site. A logo associated with this community is based on the historical headdress and named after the city of Fez, Morocco. Many fraternal organizations have adopted the use of a Fez, most notably in the United States by the Shriners organization. Throughout this book, we will tap into one of Splunk Trust's members to provide some additional best practice recommendations.
First, you will need to register for a Splunk account; go to https://www.splunk.com/. This is the account that you will use if you decide to purchase a license later. Go ahead and do this now. From here on, the password you use for your Splunk account will be referred to as your Splunk password.
- Go to the Splunk sign up page at http://www.splunk.com.
- In the upper-right corner, click on the
- Enter the information requested.
- Create a username and password.
- You will be taken to the Splunk download page, where you will need to click on the
Free Downloadbutton under Splunk Enterprise as shown in the following screenshot:
- You will then need to download the Splunk Enterprise software. Go tohttp://download.splunk.comand select the Splunk Enterprise free download. Choose your operating system, being careful to select 32 or 64-bit (whichever is appropriate in your case; most should select 64-bit, which most computers of today use). For Windows, download the
*.msifile. For Linux, download the
*.tgzfile. In this book, we work with version 7.0.1.
The installation is very straightforward. Follow the steps for your particular operating system, whether it is Windows or Linux.
- Change the installation path to
C:Splunk. You will thank us later as it simplifies issuing Splunk CLI (command line interface) commands. This is also a best practice used by modern Windows administrators. Remember to eliminate white spaces in directory names as well, as it causes complications with scripting. Click on
Nextto continue, as seen in this screenshot:
- Install Splunk Enterprise as
Local Systemand then click on
- Leave the checkbox selected to
Create Start Menu Shortcut.
- Click on
- Wait for the installation to complete.
- Click on
Finishto complete the installation. It will attempt to launch Splunk for the first time in your default browser.
Throughout the book, you will see references to
$SPLUNK_HOME. This will be the installation directory of Splunk. In Windows, as a convention used in this book,
$SPLUNK_HOME will be at
If you choose to install Splunk on a Linux machine, these are the instructions you need to follow. Take your time and do not rush the installation. For this Linux installation, the steps assume you will run Splunk in your user profile's home directory:
- Decompress the
.tgzfile you downloaded. The result of the decompression is a
splunkfolder and all the related Splunk files contained inside.
- Change the working directory to
- Run the following command to start Splunk for the first time:
Tip from the Fez: In most organizational environments, it is common to install Splunk in
/opt/. When installing on Linux, it is best practice to adjust the ulimit settings and disable transparent hugh page functionality for Splunk applications that will be loading data in real time, supporting a community of users.
Tip from the Fez: Unlike Windows installations, Splunk Linux installations aren't automatically configured to start upon reboot of the Splunk server. Ensure that you configure your Splunk instance on Linux for what is known as boot start so that if the Linux instance is restarted, Splunk will automatically restart with the operating system.
Tip from the Fez: In an enterprise environment, create a Linux user specifically for running Splunk and use that. Do not run Splunk as the Linux root user or an individuals user.
Launch the application for the first time in your default browser. You can also manually access the Splunk web page via the
http://localhost:8000 URL if you have installed locally. If using a cloud instance, use
http://ipaddress:8000 since you are connecting via a traditional internet connection as opposed to locally.
Splunk requires you to use a modern browser. It supports most versions of Google Chrome, Firefox, and newer versions of Internet Explorer. It may not support older versions of Internet Explorer.
Log in with the default username as
admin and password as
changeme, as indicated in the following screenshot:
The next step is to change the default administrator password, while keeping the default username. Do not skip this step. Make security an integral part of your day-to-day routine. Choose a password that will be secure.
Assuming that all goes well, you will now see the default Splunk Enterprise landing page as follows:
- Click directly on the green
Search & Reportingapp button. In this example, you use Splunk's very own internal index; this is Splunk's way of splunking itself (or collecting detailed information on all its underlying processes).
An index is the term given to where Splunk stores event data captured from log files, APIs, HTTP events, delimited files, and other machine data sources. It is in some ways like a database, but should not be compared to traditional relational databases in functionality or performance.
- In the
New Searchinput, type in the following search query (more about the Search Processing Language (SPL) in Chapter 3, Search Processing Language):
SPL> index=_internal sourcetype=splunkd
SPL>prefix will be used as a convention in this book to indicate a Splunk
Searchcommand. Since SPL is submitted via the Splunk user interface, there is no difference whether operating in Windows or Linux environments. The underscore before the index name
_internalmeans that it is a system index internally used by Splunk. Omitting the underscore will not yield any result, as internal is not a default index.
- This search query will have as an output the raw events from the
metrics.logfile that is stored in the
_internalindex. A log file keeps track of every event that takes place in the system. The
_internalindex keeps track of every event that occurs and makes it easily accessible.
- Take a look at these raw events, as shown in the following screenshot. You will see fields listed on the left side of the screen. The important
sourcetype. We will go into more detail about these later, but suffice to say that you will frequently search on one of these, as we have done here. As you can see from the highlighted fields, we indicated that we were looking for events where
Selected Fields, you will see
Interesting Fields. As you can tell, the purposes of many of these fields are easy to guess as seen in the following screenshot:
It is good practice to create a custom Splunk app to isolate the configurations you add to Splunk. You may never have created an app before, but you will quickly see it is not very difficult. Here, we will create a basic app called
Destinations that we will use throughout this book:
- Let's access the
Manage Appspage. There are two ways to do this; you can either click on the
Appsicon at the home page as shown in the following screenshot:
- Or, select
Manage Appsfrom the app dropdown in the top navigation bar of the
Search & Reportingapp:
- At the
Appspage, click on the
Create appicon shown in the following screenshot:
- Finally, populate the forms with the following information to complete app creation. When you are done, click on the
Savebutton to create your first Splunk app:
We will use this app to complete the exercises in this book, but first we need to make a few important changes:
- Click on the
Permissionslink shown in the preceding screenshot
- In the next window, under the
Sharing for config file-only objectssection, select
These steps will ensure that the application will be accessible to the Eventgen add-on, which will be installed later in the chapter. Use the following screenshot as a guide:
Splunk permissions are always composed of three columns:
Write. A role refers to certain authorizations or permissions that can be taken on by a user. Selecting
Read for a particular role grants the set of users in the role permission to view the object. Selecting
Write will allow the set of users to modify the object. In the preceding screenshot, everyone (all users) will have access to view the
Destinations app, but only the admin (you) and a power user can modify it.
Machine data is produced by the many functions carried out by computers and other mechanical machines. If you work in an environment that is rich in machine data, you will most likely have many sources of readily available data inputs for Splunk. However, to facilitate learning in this book, we will use a Splunk add-on called the Splunk Eventgen to easily build real-time and randomized web log data. This is the type of data that would be produced by a web-based e-commerce company.
If you need more detailed information about Eventgen, you can follow the project's GitHub repository at https://github.com/splunk/eventgen/.
Here's an important tip for Windows users. Make it a habit to always launch your Command Prompt in administrator mode. This allows you to use commands that are unhindered by Windows security:
- Right-click on the Windows start menu icon and select
Search. In Windows 7, you can click on the Windows icon and the search window will be directly above it. In Windows 10, there is a search bar named
Cortananext to the Windows icon that you can type into. They both have the same underlying function.
- In the search bar, type
- In the search results, look for
command.exe(Windows 7) or Command Prompt (Windows 10), right-click on it, and then select
Run as administrator.
Familiarize yourself with this step. Throughout the rest of the book, you will be frequently asked to open Command Prompt in administrator mode. You will know if you are in administrator mode, as it will say administrator: Command Prompt in the title of the Command Prompt window.
A Splunk add-on extends and enhances the base functionality of Splunk. An add-on typically enriches data using prebuilt rules for a specific source to enable faster analysis. In this section, you will be installing your first add-on, called Splunk Eventgen, which will help us pre-populate Splunk with real-time simulated web data:
- Download the ZIP file from the Eventgen public repository, http://github.com/splunk/eventgen. Click on the green
downloadbutton. For Linux users, we've placed a copy of the Eventgen files used in this book at our download site: https://github.com/PacktPublishing/Splunk-7-Essentials-Third-Edition.
- Extract the ZIP file to the root location for your environment.
- Rename the extracted folder to
SA-Eventgen. In Windows, this can be done with the Windows GUI, or via the
mvcommand in Linux.
- Open an administrator Command Prompt or Linux shell and execute the following command (the slashes are important):
Windows:C:> xcopy SA-Eventgen C:\Splunk\etc\apps\SA-Eventgen /O /X /E /H /K Linux:mv SA-Eventgen /$SPLUNK_HOME/etc/apps/
- In the prompt, type the following directory command to verify that the copy works properly and the contents are in the folder:
Windows: C:> dir C:\Splunk\etc\apps\SA-Eventgen Linux:ls -l /$SPLUNK_HOME/etc/apps/
These are the contents of the recently copied
SA-Eventgen folder, as shown in the following (Windows) screenshot, the same as what would appear in the similar location on Linux:
Return to the Splunk interface in your web browser and restart Splunk by selecting the
Settingsdropdown; under the
SYSTEMsection, click on
- On the
Server controlspage, click on the
Restart Splunkbutton. Click on
OKwhen asked to confirm the restart.
- The web interface will first notify you that Splunk is restarting in the background, then it will tell you that the restart has been successful. Every time Splunk is restarted, you will be prompted to log in with your credentials. Go ahead and log in.
- After logging in, you will now see the Eventgen app in the Splunk Enterprise landing page. Go to the
Manage Appspage and confirm that the
SA-EventGenapplication is installed:
You have successfully installed a Splunk add-on.
There are several different ways to stop, start, or restart Splunk. The easiest way to restart Splunk is to do it from the web interface, as demonstrated in the preceding section. The web interface, however, only allows you to restart your Splunk instance. It does not offer any other control options.
The most flexible way to control Splunk is by using the CLI. Using the CLI is an essential skill for Splunk administrators.
In the console or Command Prompt, type in the following command and hit Enter on your keyboard:
Windows: C:> cd C:Splunk\bin Linux: cd /$SPLUNK_HOME/bin
While in the
$SPLUNK_HOME/bin directory, issue the following command to restart Splunk:
Windows: C:Splunk\bin>splunk restart Linux: [[email protected] bin]$ ./splunk restart
After issuing this command,
splunkd will go through its restart process. Here are the other basic parameters that you can pass to the Splunk application to control Splunk:
splunk status: Tells you whether
splunkdis running or not
splunk stop: Stops
splunkdand all its processes
splunk start: Starts
splunkdand all its processes
splunk restart: Restarts
splunkdand all its processes
./splunk status: Tells you whether
splunkdis running or not
./splunk stop: Stops
splunkdand all its processes
./splunk start: Starts
splunkdand all its processes
./splunk restart: Restarts
splunkdand all its processes
Doing this from a CLI gives the added benefit of verbose messages. A verbose message is a message with a lot of information in it. Such messages can be useful for making sure the system is working correctly or troubleshooting any errors.
A successful restart of
splunkd generally has the following output (elements of which may vary):
In Windows, you can also control Splunk through the
Splunkd Service, as shown in the following screenshot. The
d in the service name, denoting daemon, means a background process. Note that the second service,
splunkweb, is not running. Do not try to start
splunkweb as it is deprecated and is there only for legacy purposes. The Splunk-running web application is now bundled in
We are almost there. Proceed by first downloading the exercise materials that will be used in this book. Open an administrator Command Prompt and make sure you are in the root of the Windows machine or Linux user shell. Download the ZIP file and extract it in your computer using https://github.com/PacktPublishing/Splunk-7-Essentials-Third-Edition.
The Eventgen configuration you will need for the exercises in this book has been packaged and is ready to go. We are not going into the details of how to configure Eventgen. If you are interested in learning more about Eventgen, visit the project page at http://github.com/splunk/eventgen.
Follow these instructions to proceed:
- Extract the project ZIP file into your local machine. Open an administrator console and use the change directory command to set where you extracted the file.
- Create a new
samplesdirectory in the
DestinationsSplunk app. The path of this new directory will be
Windows:C:> mkdir C:\Splunk\etc\apps\destinations\samples Linux:mkdir /splunk/etc/apps/destinations/samples
- Copy all the
/labs/chapter01/eventgenof the extracted project directory into the newly created
samplesdirectory. Windows users can also copy and paste using a GUI:
Windows:C:> copy C:\splunk-essentials- master\labs\chapter01\eventgen*.sample C:\Splunk\etc\apps\destinations\samples Linux: cp /splunk-essentials- master/labs/chapter01/eventgen/*.sample /splunk/etc/apps/destinations/samples
- Now, copy the
$SPLUNK_HOME/etc/apps/destinations/localdirectory. Windows users can also copy and paste using the GUI if you prefer:
Windows: C:> copy C:\splunk-essentials-master\labs\ chapter01\eventgen\eventgen.conf C:\Splunk\etc\apps\destinations\local Linux:cp /splunk-essentials-master/labs/chapter01/eventgen.conf /splunk/etc/apps/destinations/local
- Grant the
SYSTEMWindows account full access permissions to the
eventgen.conffile. This is a very important step. You can either do it using the following
icaclscommand or change it using the Windows GUI. This step (Step 5) is not required for Linux users, who can move on to Step 6:
C:> icacls C:\Splunk\etc\apps\destinations\local\eventgen.conf /grant SYSTEM:F
A successful output of this command will look like this:
processed file: C:\Splunk\etc\apps\destinations\local\eventgen.confSuccessfully processed 1 files; Failed processing 0 files
- Restart Splunk.
Next we will see our
Destinations app in action! Remember that we have configured it to draw events from a prototype web company. That is what we did when we set it up to work with Eventgen. Now, let's look at some of our data:
- After a successful restart, log back in to Splunk and proceed to your new
- In the
Searchfield, type this search query and select Enter:
Examine the event data that your new app is enabling to come into Splunk. You will see a lot of references to browsers, systems, and so forth, the kinds of information that make a web-based e-commerce company run.
Try changing the time range to
5 minute window) to see the data flow in before your eyes:
Congratulations! You now have real-time web log data that we can use in subsequent chapters.
Tip from the Fez: Running a Splunk report under a real-time window places heavier strain on Splunk because it is rerunning the search over and over to generate the live nature of the real-time window. Unless absolutely needed, choose to have reports run for a set time period on user demand or a previously assigned schedule.
Now that we have data in Splunk, it's time to use it in order to derive something meaningful out of it. You are still in the
Destinations app, correct? We will show you the basic routine when creating new dashboards and dashboard panels.
Type in or copy/paste the following search query in the
Search Field, then hit Enter:
SPL> index=main /booking/confirmation [email protected] | timechart count span=15m
After the search results render, click on the
Visualization tab. This will switch your view into a visualization so you can readily see how your data will look. By default, it should already be using the
Column Chart as shown in the following screenshot. If it does not, then use the screenshot as a guide on how to set it:
Now that you can see your
Column Chart, it is time to save it as a dashboard. Click on
Save As in the upper-right corner of the page, then select
Dashboard Panel, as shown in the following screenshot:
Now, let's fill up that dashboard panel information, as seen in the following screenshot. Make sure to select the
Shared in App in the
Dashboard Permissions section:
Finish up by clicking
View Dashboard in the next prompt:
You have created your very first Splunk dashboard with a single panel telling you the number of confirmed bookings in the last 24 hours at 15-minute intervals. If you let Eventgen run for a while and rerun the dashboard, the contents will populate as the data is generated. Time to show it to your boss!
In one chapter we acquired, installed, and used Splunk. We now have a fully-functional Splunk installation with live data. Leave Splunk running for two hours or so. After a few hours, you can stop Splunk if you need to rest for a bit to suppress indexing and restart it when you're ready to proceed into the next chapters.
Do you recall how to control Splunk from the command line?
Windows: C:Splunkbin> splunk stopC:Splunkbin> splunk start C:Splunkbin> splunk restart Linux: bin]$ ./splunk stop bin]$ ./splunk start bin]$ ./splunk restart
In this chapter, you learned a number of basic Splunk concepts that you need to get started with this powerful tool. You learned how to install Splunk and configure a new Splunk app. You ran a simple search against the Splunk internal logs to ensure that the application is functional. You then installed a Splunk add-on called Eventgen, which you used to populate dummy data into Splunk in real time. You were shown how to control Splunk using the web user interface and the CLI. Finally, you created your very first Splunk dashboard, based off a single search. We will get a lot more creative with dashboards in additional chapters later in this book.
Now, we will go on to Chapter 2, Bringing in Data, to learn more about how to input data.