Implementing Splunk (Update)

Chapter 1. The Splunk Interface

This chapter will walk you through the most common elements in the Splunk interface, and will touch upon concepts that will be covered in greater detail, in later chapters. You may want to dive right into the search section, but an overview of the user interface elements might save you some frustration later. We will cover the following topics in this chapter:

Logging in and app selection
A detailed explanation of the search interface widgets
A quick overview of the admin interface

The home app

After logging in, the default app is the Launcher app (some may refer to this as Home). This app is a launching pad for apps and tutorials.

In earlier versions of Splunk, the Welcome tab provided two important shortcuts, Add data and the Launch search app. In version 6.2.0, the Home app is divided into distinct areas, or panes, that provide easy access to Explore Splunk Enterprise (Add Data, Splunk Apps, Splunk Docs, and Splunk Answers) as well as Apps (the App management page) Search & Reporting (the link to the Search app), and an area where you can set your default dashboard (choose a home dashboard). We'll cover apps & dashboards in later chapters of this book.

The Explore Splunk Enterprise pane shows links to:

Add data: This links Add Data to the Splunk page. This interface is a great start for getting local data flowing into Splunk (making it available to Splunk users). The Preview data interface takes an enormous amount of complexity out of configuring dates and line breaking. We won't go through those interfaces here, but we will go through the configuration files that these wizards produce in Chapter 11, Configuring Splunk.
Splunk Apps: This allows you to find and install more apps from the Splunk Apps Marketplace (http://apps.splunk.com). This marketplace is a useful resource where Splunk users and employees post Splunk apps, mostly free but some premium ones as well.
Splunk Answers: This is one of your links to the wide amount of Splunk documentation available, specifically http://answers.splunk.com, where you can engage with the Splunk community on Splunkbase (https://splunkbase.splunk.com/) and learn how to get the most out of your Splunk deployment.

The Apps section shows the apps that have GUI elements on your instance of Splunk. App is an overloaded term in Splunk. An app doesn't necessarily have a GUI at all; it is simply a collection of configurations wrapped into a directory structure that means something to Splunk. We will discuss apps in a more detailed manner in Chapter 8, Working with Apps.

Search & Reporting is the link to the Splunk Search & Reporting app.

Beneath the Search & Reporting link, Splunk provides an outline which, when you hover over it, displays a Find More Apps balloon tip. Clicking on the link opens the same Browse more apps page as the Splunk Apps link mentioned earlier.

Choose a home dashboard provides an intuitive way to select an existing (simple XML) dashboard and set it as part of your Splunk Welcome or Home page. This sets you at a familiar starting point each time you enter Splunk. The following image displays the Choose Default Dashboard dialog:

Once you select an existing dashboard from the dropdown list, it will be part of your welcome screen every time you log into Splunk – until you change it. There are no dashboards installed by default after installing Splunk, except the Search & Reporting app. Once you have created additional dashboards, they can be selected as the default.

The top bar

The bar across the top of the window contains information about where you are, as well as quick links to preferences, other apps, and administration.

The current app is specified in the upper-left corner. The following image shows the upper-left Splunk bar when using the Search & Reporting app:

Clicking on the text takes you to the default page for that app. In most apps, the text next to the logo is simply changed, but the whole block can be customized with logos and alternate text by modifying the app's CSS. We will cover this in Chapter 8, Working with Apps.

The upper-right corner of the window, as seen in the previous image, contains action links that are almost always available:

The name of the user who is currently logged in appears first. In this case, the user is Administrator. Clicking on the username allows you to select Edit Account (which will take you to the Your account page) or to Logout (of Splunk). Logout ends the session and forces the user to login again. The following screenshot shows what the Your account page looks like:
This form presents the global preferences that a user is allowed to change. Other settings that affect users are configured through permissions on objects and settings on roles. (Note: preferences can also be configured using the CLI or by modifying specific Splunk configuration files).
Full name and Email address are stored for the administrator's convenience.
Time zone can be changed for the logged-in user. This is a new feature in Splunk 4.3.
Note
Setting the time zone only affects the time zone used to display the data. It is very important that the date is parsed properly when events are indexed. We will discuss this in detail in Chapter 2, Understanding Search.
Default app controls the starting page after login. Most users will want to change this to search.
Restart backgrounded jobs controls whether unfinished queries should run again if Splunk is restarted.
Set password allows you to change your password. This is only relevant if Splunk is configured to use internal authentication. For instance, if the system is configured to use Windows Active Directory via LDAP (a very common configuration), users must change their password in Windows.
Messages allows you to view any system-level error messages you may have pending. When there is a new message for you to review, a notification displays as a count next to the Messages menu. You can click the X to remove a message.
Tip
Downloading the example code
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
The Settings link presents the user with the configuration pages for all Splunk Knowledge objects, Distributed Environment settings, System and Licensing, Data, and Users and Authentication settings. If you do not see some of these options, you do not have the permissions to view or edit them.
The Activity menu lists shortcuts to Splunk Jobs, Triggered Alerts, and System Activity views. You can click Jobs (to open the search jobs manager window, where you can view and manage currently running searches), click Triggered Alerts (to view scheduled alerts that are triggered) or click System Activity (to see dashboards about user activity and the status of the system).
Help lists links to video Tutorials, Splunk Answers, the Splunk Contact Support portal, and online Documentation.
Find can be used to search for objects within your Splunk Enterprise instance. For example, if you type in error, it returns the saved objects that contain the term error. These saved objects include Reports, Dashboards, Alerts, and so on. You can also search for error in the Search & Reporting app by clicking Open error in search.

The search & reporting app

The Search & Reporting app (or just the search app) is where most actions in Splunk start. This app is a dashboard where you will begin your searching.

The data generator

If you want to follow the examples that appear in the next few chapters, install the ImplementingSplunkDataGenerator demo app by following these steps:

Download ImplementingSplunkDataGenerator.tar.gz from the code bundle available on the site http://www.packtpub.com/support.
Choose Manage apps… from the Apps menu.
Click on the button labeled Install app from file.
Click on Choose File, select the file, and then click on Upload.

This data generator app will produce about 16 megabytes of output per day. The app can be disabled so that it stops producing data by using Manage apps…, under the App menu.

The summary view

Within the Search & Reporting app, the user is presented with the Summary view, which contains information about the data which that user searches for by default. This is an important distinction—in a mature Splunk installation, not all users will always search all data by default. But at first, if this is your first trip into Search & Reporting, you'll see the following:

From the screen depicted in the previous screenshot, you can access the Splunk documentation related to What to Search and How to Search. Once you have at least some data indexed (a topic we'll discuss later in the The settings section), Splunk will provide some statistics on the available data under What to Search (remember that this reflects only the indexes that this particular user searches by default; there are other events that are indexed by Splunk, including events that Splunk indexes about itself. We will discuss indexes in Chapter 9, Building Advanced Dashboards). This is seen in the following image:

In previous versions of Splunk, panels such as the All indexed data panel provided statistics for a user's indexed data. Other panels gave a breakdown of data using three important pieces of metadata—Source, Sourcetype, and Hosts. In the current version—6.2.0—you access this information by clicking on the button labeled Data Summary, which presents the following to the user:

This dialog splits the information into three tabs—Hosts, Sources and Sourcetypes.

A host is a captured hostname for an event. In the majority of cases, the host field is set to the name of the machine where the data originated. There are cases where this is not known, so the host can also be configured arbitrarily.
A source in Splunk is a unique path or name. In a large installation, there may be thousands of machines submitting data, but all data on the same path across these machines counts as one source. When the data source is not a file, the value of the source can be arbitrary, for instance, the name of a script or network port.
A source type is an arbitrary categorization of events. There may be many sources across many hosts, in the same source type. For instance, given the sources /var/log/access.2012-03-01.log and /var/log/access.2012-03-02.log on the hosts fred and wilma, you could reference all these logs with source type access or any other name that you like.

Let's move on now and discuss each of the Splunk widgets (just below the app name). The first widget is the navigation bar.

As a general rule, within Splunk, items with downward triangles are menus. Items without a downward triangle are links.

We will cover customizing the navigation bar in Chapter 8, Working with Apps.

Next we find the Search bar. This is where the magic starts. We'll go into great detail shortly.

Search

Okay, we've finally made it to search. This is where the real power of Splunk lies.

For our first search, we will search for the word (not case specific); error. Click in the search bar, type the word error, and then either press Enter or click on the magnifying glass to the right of the bar.

Upon initiating the search, we are taken to the search results page.

Note

Note that the search we just executed was across All time (by default); to change the search time, you can utilize the Splunk time picker.

Refer to the Using the time picker section for details on changing the time frame of your search.

Actions

Let's inspect the elements on this page. Below the Search bar, we have the event count, action icons, and menus.

Starting from the left, we have the following:

The number of events matched by the base search. Technically, this may not be the number of results pulled from disk, depending on your search. Also, if your query uses commands, this number may not match what is shown in the event listing.
Job: This opens the Search job inspector window, which provides very detailed information about the query that was run.
Pause: This causes the current search to stop locating events but keeps the job open. This is useful if you want to inspect the current results to determine whether you want to continue a long running search.
Stop: This stops the execution of the current search but keeps the results generated so far. This is useful when you have found enough and want to inspect or share the results found so far.
Share: This shares the search job. This option extends the job's lifetime to seven days and sets the read permissions to everyone.
Export: This exports the results. Select this option to output to CSV, raw events, XML, or JavaScript Object Notation (JSON) and specify the number of results to export.
Print: This formats the page for printing and instructs the browser to print.
Smart Mode: This controls the search experience. You can set it to speed up searches by cutting down on the event data it returns and, additionally, by reducing the number of fields that Splunk will extract by default from the data (Fast mode). You can, otherwise, set it to return as much event information as possible (Verbose mode). In Smart mode (the default setting) it toggles search behavior based on the type of search you're running.

Timeline

Now we'll skip to the timeline below the action icons.

Along with providing a quick overview of the event distribution over a period of time, the timeline is also a very useful tool for selecting sections of time. Placing the pointer over the timeline displays a pop-up for the number of events in that slice of time. Clicking on the timeline selects the events for a particular slice of time.

Clicking and dragging selects a range of time.

Once you have selected a period of time, clicking on Zoom to selection changes the time frame and reruns the search for that specific slice of time. Repeating this process is an effective way to drill down to specific events.

Deselect shows all events for the time range selected in the time picker.

Zoom out changes the window of time to a larger period around the events in the current time frame

The field picker

To the left of the search results, we find the field picker. This is a great tool for discovering patterns and filtering search results.

Fields

The field list contains two lists:

Selected Fields, which have their values displayed under the search event in the search results
Interesting Fields, which are other fields that Splunk has picked out for you

Above the field list are two links: Hide Fields and All Fields.

Hide Fields: Hides the field list area from view.
All Fields: Takes you to the Selected Fields window.

Search results

We are almost through with all the widgets on the page. We still have a number of items to cover in the search results section though, just to be thorough.

As you can see in the previous screenshot, at the top of this section, we have the number of events displayed. When viewing all results in their raw form, this number will match the number above the timeline. This value can be changed either by making a selection on the timeline or by using other search commands.

Next, we have the action icons (described earlier) that affect these particular results.

Under the action icons, we have four results tabs:

Events list, which will show the raw events. This is the default view when running a simple search, as we have done so far.
Patterns streamlines the event pattern detection. It displays a list of the most common patterns among the set of events returned by your search. Each of these patterns represents the number of events that share a similar structure.
Statistics populates when you run a search with transforming commands such as stats, top, chart, and so on. The previous keyword search for error does not display any results in this tab because it does not have any transforming commands.
Visualization transforms searches and also populates the Visualization tab. The results area of the Visualization tab includes a chart and the statistics table used to generate the chart. Not all searches are eligible for visualization – a concept which will be covered later in Chapter 3, Tables, Charts, and Fields.

Under the tabs described just now, is the timeline which we will cover in more detail later in this chapter in the section Using the time picker.

Options

Beneath the timeline, (starting at the left) is a row of option links that include:

Show Fields: shows the Selected Fields screen
List: allows you to select an output option (Raw, List, or Table) for displaying the search results
Format: provides the ability to set Result display options, such as Show row numbers, Wrap results, the Max lines (to display) and Drilldown as on or off.
NN Per Page: is where you can indicate the number of results to show per page (10, 20, or 50).

To the right are options that you can use to choose a page of results, and to change the number of events per page.

Note

In prior versions of Splunk, these options were available from the Results display options popup dialog.

The events viewer

Finally, we make it to the actual events. Let's examine a single event.

Starting at the left, we have:

Event Details: Clicking here (indicated by the right facing arrow) opens the selected event, providing specific information about the event by type, field, and value, and allows you the ability to perform specific actions on a particular event field. In addition, Splunk version 6.2.0 offers a button labeled Event Actions to access workflow actions, a few of which are always available.
Build Eventtype: Event types are a way to name events that match a certain query. We will dive into event types in Chapter 7, Extending Search.
Extract Fields: This launches an interface for creating custom field extractions. We will cover field extraction in Chapter 3, Tables, Charts, and Fields.
Show Source: This pops up a window with a simulated view of the original source.
The event number: Raw search results are always returned in the order most recent first.
Next to appear are any workflow actions that have been configured. Workflow actions let you create new searches or links to other sites, using data from an event. We will discuss workflow actions in Chapter 7, Extending Search.
Next comes the parsed date from this event, displayed in the time zone selected by the user. This is an important and often confusing distinction. In most installations, everything is in one time zone—the servers, the user, and the events. When one of these three things is not in the same time zone as the others, things can get confusing. We will discuss time in great detail in Chapter 2, Understanding Search.
Next, we see the raw event itself. This is what Splunk saw as an event. With no help, Splunk can do a good job finding the date and breaking lines appropriately, but as we will see later, with a little help, event parsing can be more reliable and more efficient.
Below the event are the fields that were selected in the field picker. Clicking on the value adds the field value to the search.

The settings section

The Settings section, in a nutshell, is an interface for managing configuration files. The number of files and options in these configuration files is truly daunting, so the web interface concentrates on the most commonly used options across the different configuration types.

Splunk is controlled exclusively by plain text configuration files. Feel free to take a look at the configuration files that are being modified as you make changes in the admin interface. You will find them by hitting the following commands:

$SPLUNK_HOME/etc/system/local/
$SPLUNK_HOME/etc/apps/

You may notice configuration files with the same name at different locations. We will cover in detail; the different configuration files, their purposes, and how these configurations merge together in Chapter 11, Configuring Splunk. Don't start modifying the configurations directly until you understand what they do and how they merge.

Clicking on Settings, on the top bar, takes you to the Settings page.

The options are organized into logical groupings, as follows:

Knowledge: Each of the links under Knowledge allows you to control one of the many object types that are used at search time. The following screenshot shows an example of one object type, workflow actions: Searches, reports, and alerts:
System: The options under this section control system-wide settings.
- System settings covers network settings, the default location to store indexes, outbound e-mail server settings, and how much data Splunk logs about itself
- Server controls contains a single page that lets you restart Splunk from the web interface
- Licensing lets you add license files or configure Splunk as a slave to a Splunk license server
Data: This section is where you manage data flow.
- Data Inputs: Splunk can receive data by reading files (either in batch mode or in real time), listening to network ports, or running scripts.
- Forwarding and receiving: Splunk instances don't typically stand alone. Most installations consist of at least one Splunk indexer and many Splunk forwarders. Using this interface, you can configure each side of this relationship and more complicated setups (we will discuss this in more detail in Chapter 12, Advanced Deployments).
- Indexes: An Index is essentially a datastore. Under the covers, it is simply a set of directories, created and managed by Splunk. For small installations, a single index is usually acceptable. For larger installations, using multiple indexes allows flexibility in security, retention, and performance tuning, as well as better use of hardware. We will discuss this further in Chapter 11, Configuring Splunk.
- Report acceleration summaries: Accesses automatically-created summaries to speed up completion times for certain kinds of reports.
Distributed Environment: The three options here relate to distributed deployments (we will cover these options in detail in Chapter 12, Advanced Deployments):
- Indexer clustering: Access to enabling and configuring Splunk Indexer clustering, which we will discuss later in Chapter 10, Summary Indexes and CSV Files.
- Forwarder management: Access to the forwarder management UI distributes deployment apps to Splunk clients.
- Distributed search: Any Splunk instance running searches can utilize itself and other Splunk instances to retrieve results. This interface allows you to configure access to other Splunk instances.
Users and authentication: This section provides authentication controls and an account link.
- Access controls: This section is for controlling how Splunk authenticates users and what users are allowed to see and do. We will discuss this further in Chapter 11, Configuring Splunk.
- In addition to the links, the Settings page also presents a panel on the left side of the page. This panel includes two icons Add Data and Distributed Management Console.
- Add Data links to the Add Data page. This page presents you with three options for getting data into your Splunk Enterprise instance: Upload, Monitor, and Forward.
- Distributed Management Console, where you can view detailed performance information about your Splunk Enterprise deployment.

Ed P Aug 20, 2015

This massive volume (clocking in at over 450 pages!) is easily one of the better IT books I've read in the past few years. Although its called Implementing Splunk it DOES NOT cover installing the software. Check the documentation for that. What it does cover is getting started with using the software, and working towards more in-depth scenarios and configurations so you can handle a wider variety of data, with better dashboards and visualizations. I think one of the best parts of this book is the number of screenshots with clearly marked emphasis on the areas that the author is referring to in the text. That in itself makes following the text much easier, since you aren't hunting around trying to find the setting, menu or piece of information, but you can immediately see it. Need touch!

Amazon Verified review

Ishaan Shekhar Aug 10, 2015

Very nice book! Much better than the first edition. Good for beginners and medium-pros.

Gary Jun 10, 2017

Good starter book

SM Lawson Aug 19, 2015

First and foremost let me say I enjoyed the first book and was very excited to purchase this book on theexpectations that this would surpass it. I was not disappointed the book delves into the extensive concepts ofBig Data and intelligence gathering concepts used in the real world, that said I have one extensive criticism and thatwould be the instructions given on setting up the tutorial data. The Event generator is not really windows friendly and instructionsare not quite clear on setting it up... the instructions given do not work within windows as given and I suppose were only tested in a Linuxenvironment. It would help immensely if the Writers could address this issue with a solution for windows installs concerning the event generator propagated by Splunk making sure step by step instructions are given to make the data work with Splunk. Much easier to work with examples in book.

Implementing Splunk (Update): A comprehensive guide to help you transform Big Data into valuable business insights with Splunk 6.2

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Note

Tip

Fields

Options

The events viewer

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the authors

FAQs

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access