Chapter 6. Extending Search

In this chapter, we will look at some of the features that Splunk provides to go beyond its already powerful search language. We will cover the following with the help of examples:

• Tags and event types that help you categorize events, both for search and reporting

• Lookups that allow you to add external fields to events as though they were part of the original data

• Macros that let you reuse snippets of search in powerful ways

• Workflow actions that let you build searches and links based on field values in an event

• External commands that allow you to use Python code to work with search results

In this chapter, we will investigate a few of the many commands included in Splunk. We will write our own commands in Chapter 12, Extending Splunk.

Using tags to simplify search

Tags allow you to attach a "marker" to fields and event types in Splunk. You can then search and report on these tags later. Let's attach a tag to a couple of users who are administrators. Start with the following search:

sourcetype="impl_splunk_gen"
  | top user

This search gives us a list of our users such as mary, linda, Bobby, jacky, bob, and extrauser:

Let's say that in our group, linda and jacky are administrators. Using a standard search, we can simply search for these two users like this:

sourcetype="impl_splunk_gen" (user=linda OR user=jacky)

Searching for these two users while going forward will still work, but instead if we search for the tag value, we can avoid being forced to update multiple saved queries in the future.

To create a tag, first we need to locate the field.

If the user field isn't already visible, click on it in the field picker, and then click on Select and show in results:

With the menu now visible, we can tag this value of the user field...

Using event types to categorize results

An event type is essentially a simple search definition, with no pipes or commands. To define an event type, first make a search. Let's search for:

sourcetype="impl_splunk_gen" logger="AuthClass"

Let's say these events are login events. To make an event type, choose Event type... from the Create menu, as shown here:

This presents us with a dialog, where we can assign a Name string and optionally any Tags(s) to this event type, as shown in the following screenshot:

Let's name our event type login.

We can now search for the same events using the event type:

eventtype=login

Event types can be used as part of another search, as follows:

eventtype=login loglevel=error

Event type definitions can also refer to other event types. For example, let's assume that all login events that have a loglevel value of ERROR are in fact failed logins.

We can now save this into another event type using the same steps as mentioned previously. Let's call it failed_login. We can now...

Using lookups to enrich data

Sometimes, information that would be useful for reporting and searching is not located in the logs themselves, but is available elsewhere. Lookups allow us to enrich data, and even search against the fields in the lookup as if they were part of the original events.

The source of data for a lookup can be either a Comma Separated Values (CSV) file or a script. We will cover the most common use of a CSV lookup in the next section. We will cover scripted lookups in Chapter 12, Extending Splunk.

There are three steps for fully defining a lookup: creating the file, defining the lookup definition, and optionally wiring the lookup to run automatically.

Defining a lookup table file

A lookup table file is simply a CSV file. The first line is treated as a list of field names for all other lines.

Lookup table files are managed at Manager | Lookups | Lookup table files. Simply upload a new file and give it a filename, preferably ending in .csv.

The lookup file users.csv is included...

Using macros to reuse logic

A macro serves the purpose of replacing bits of search language with expanded phrases. Using macros can help you reuse logic and greatly reduce the length of queries.

Let's use one of our examples from Chapter 5, Advanced Search Examples, as our example case:

sourcetype="impl_splunk_web" user=mary 
  | transaction maxpause=5m user
  | stats avg(duration) avg(eventcount)

Creating a simple macro

Let's take the last two lines of our query and convert them to a macro. First, navigate to Manager | Advanced search | Advanced search | Search macros and click on New.

Walking through our fields, we have:

• Destination app: This is where the macro will live.

• Name: This is the name we will use in our searches.

• Definition: This is the text that will be placed in our search.

• Use eval-based definition?: If checked, the Definition string is treated as an eval statement instead of raw text. We'll use this option later.

• The remaining fields are used if arguments are specified. We will...

Creating workflow actions

Workflow actions allow us to create custom actions based on the values in search results. The two supported actions either run a search or link to a URL.

Running a new search using values from an event

To build a workflow action, navigate to Manager | Fields | Workflow actions and click on New. You are presented with this form:

Let's walk through the following fields:

• Destination app: This is the app where the workflow action definition will live.

• Name: This is the name used in configuration files. This name cannot contain spaces, but underscores are fine.

• Label: This is what will appear in the menu. It can contain variables. In this case, we have included $user$, which will be populated with the value of the user field from the event.

• Apply only to the following fields: This workflow action will only appear on an event if all fields specified in this list have a value. Show action in will determine which menus can contain the workflow action.

• Apply only to the following...

Using external commands

The Splunk search language is extremely powerful, but at times, it may be either difficult or impossible to accomplish some piece of logic by using nothing but the search language. To deal with this, Splunk allows external commands to be written in Python. A number of commands ship with the product, and a number of commands are available in apps at http://splunk-base.splunk.com/.

Let's try out a few of the included commands. The documentation for the commands is included with other search commands at http://docs.splunk.com/. You can find a list of all included commands, both internal and external, by searching for All search commands. We will write our own commands in Chapter 12, Extending Splunk.

Extracting values from XML

Fairly often, machine data is written in XML format. Splunk will index this data without any issue, but it has no native support for XML. Though XML is not an ideal logging format, it can usually be parsed simply enough. Two commands are included...

Summary

In this chapter, we quickly covered tags, event types, lookups, macros, workflow actions, and external commands. I hope these examples and discussions will serve as starting points for your apps. More examples can be found in the official Splunk documentation at http://docs.splunk.com/ and at http://splunk-base.splunk.com/.

In the next chapter, we will dive into creating and customizing our own apps.