Chapter 10. Configuring Splunk

Everything that controls Splunk lives in configuration files sitting in the filesystem of each instance of Splunk. These files are unencrypted, easily readable, and easily editable. Almost all of the work that we have done so far has been accomplished through the web interface, but everything actually ends up in these configuration files.

While the web interface does a lot, there are many options that are not represented in the admin interface. There are also some things that are simply easier to accomplish by editing the files directly.

In this chapter, we will cover:

• Locating configuration files

• Merging configurations

• Debugging configurations

• Common configurations and their parameters

Locating Splunk configuration files

Splunk's configuration files live in $SPLUNK_HOME/etc. This is reminiscent of Unix's /etc directory but is instead contained within Splunk's directory structure. This has the advantage that the files don't have to be owned by root. In fact, the entire Splunk installation can run as an unprivileged user (assuming you don't need to open a port below 1024 or read files only readable by another user).

The directories that contain configurations are:

• $SPLUNK_HOME/etc/system/default: The default configuration files that ship with Splunk. Never edit these files as they will be overwritten each time you upgrade.

• $SPLUNK_HOME/etc/system/local: This is the location of global configuration overrides specific to this host. There are very few configurations that need to live here—most configurations that do live here are created by Splunk itself. In almost all cases, you should make your configuration files inside of an app.

• $SPLUNK_HOME/etc/apps/$app_name/default...

The structure of a Splunk configuration file

The .conf files used by Splunk look very similar to .ini files. A simple configuration looks like this:

#settings for foo
[foo]
bar=1
la = 2

Let's look at the following couple of definitions:

• stanza: A stanza is used to group attributes. Our stanza in this example is [foo]. A common synonym for this is section. Keep in mind the following key points:

  • A stanza name must be unique in a single file

  • Order does not matter

• attribute: An attribute is a name-value pair. Our attributes in this example are bar and la. A common synonym is parameter. Keep in mind the following key points:

  • The attribute name must not contain whitespace or the equals sign

  • Each attribute belongs to the stanza defined above; if the attribute appears above all stanzas, the attribute belongs to the stanza [default]

  • The attribute name must be unique in a single stanza but not in a configuration

  • Each attribute must have its own line and can only use one line

  • Spaces around the equal sign...

Configuration merging logic

Configurations in different locations merge behind the scenes into one "super" configuration. Luckily, the merging happens in a predictable way and is fairly easy to learn, and there is a tool to help us preview this merging.

Merging order

Merging order is slightly different depending on whether the configuration is being used by the search engine or another part of Splunk. The difference is whether there is an active user and app.

Merging order outside of search

Configurations being used outside of search are merged in a fairly simple order. These configurations include what files to read, what indexed fields to create, what indexes exist, and deployment server and client configurations as well as other settings. These configurations merge in this order:

1. $SPLUNK_HOME/etc/system/default: This directory contains the base configurations that ship with Splunk.

   Note

   Never make changes in $SPLUNK_HOME/etc/system/default as your changes will be lost when you upgrade Splunk...

An overview of Splunk .conf files

If you have spent any time in the filesystem investigating Splunk, you have seen many different files ending in .conf. In this section, we will give a quick overview of the most common .conf files. The official documentation is the best place to look for a complete reference of files and attributes.

Note

The quickest way to find the official documentation is with your favorite search engine by searching for splunk filename.conf. For example, a search for splunk props.conf pulls up the Splunk documentation for props.conf first in every search engine I tested.

props.conf

The stanzas in props.conf define which events to match based on host, source, and sourcetype. These stanzas are merged into the master configuration based on the uniqueness of stanza and attribute names, as with any other configuration, but there are specific rules governing when each stanza is applied to an event and in what order. Stated as simply as possible, attributes are sorted by type...

User interface resources

Most Splunk apps consist mainly of resources for the web application. The app layout for these resources is completely different from all other configurations

Views and navigation

Like .conf files, view and navigation documents take precedence in the following order:

1. $SPLUNK_HOME/etc/users/$username/$appname/local: When a new dashboard is created, it lands here. It will remain here until the permissions are changed to App or Global.

2. $SPLUNK_HOME/etc/apps/$appname/local: Once a document is shared, it will be moved to this directory.

3. $SPLUNK_HOME/etc/apps/$appname/default: Documents can only be placed here manually. You should do this if you are going to share an app.

Unlike .conf files, these documents do not merge.

Within each of these directories, views and navigation end up under the directories data/ui/views and data/ui/nav, respectively. So, given a view foo, for the user bob, in the app app1, the initial location for the document will be:

$SPLUNK_HOME/etc/users...

Summary

This chapter provided an overview of how configurations work and a commentary on the most common aspects of Splunk configuration. This is by no means a complete reference for these configurations, which I will leave to the official documentation. I find the easiest way to get to the official documentation for a particular file is to query your favorite search engine for splunk configname.conf.

In Chapter 11, Advanced Deployments, we will dig into distributed deployments, and look at how they are efficiently configured. What you have learned in this chapter will be vital to understanding what is considered best practice.