Chapter 2. Understanding Search

To successfully use Splunk, it is vital that you write effective searches. Using the index efficiently will make your initial discoveries faster, and the reports you create will run faster for you and others. In this chapter, we will cover:

• How to write effective searches

• How to search using fields

• Understanding time

• Saving and sharing searches

Using search terms effectively

The key to creating an effective search is to take advantage of the index. Splunk's index is effectively a huge word index, sliced by time. The single most important factor for the performance of your searches is how many events are pulled from disk. The following few key points should be committed to memory:

• Search terms are case insensitive: Searches for error, Error, ERROR, and ErRoR are all the same thing.

• Search terms are additive: Given the search item mary error, only events that contain both words will be found. There are Boolean and grouping operators to change this behavior; we will discuss these later.

• Only the time frame specified is queried: This may seem obvious, but it's a big difference from a database, which would always have a single index across all events in a table. Since each index is sliced into new buckets over time, only the buckets that contain events for the time frame in question need to be queried.

• Search terms are words, not parts...

Boolean and grouping operators

There are a few operators that you can use to refine your searches (note that these operators must be in uppercase to not be considered search terms):

• AND is implied between terms. error mary is the same as error AND mary.

• OR allows you to specify multiple values. error OR mary means "find any event that contains either word".

• NOT applies to the next term or group. error NOT mary would find events that contain error but do not contain mary.

• "" identifies a phrase. "Out of this world" will find this exact sequence of words. Out of this world would find any event that contains all of these words, but not necessarily in that order.

• ( ) is used for grouping terms. Parentheses can help avoid confusion in logic. For instance, these two statements are equivalent:

  • bob error OR warn NOT debug

  • (bob AND (error OR warn)) AND NOT debug

• = is reserved for specifying fields. Searching for an equal sign can be accomplished by wrapping it in quotes.

• [ ] is used to perform...

Clicking to modify your search

Though you can probably figure it out by just clicking around, it is worth discussing the behavior of the GUI when moving your mouse around and clicking.

• Clicking on any word or field value will add that term to the search.

• Clicking on a word or field value that is already in the query will remove it from the query.

• Clicking on any word or field value while holding down Alt (option on the Mac) will append that search term to the query, preceded by NOT. This is a very handy way to remove irrelevant results from query results.

Event segmentation

In Chapter 1, The Splunk Interface, we touched upon this setting in the Options dialog. The different options change what is highlighted as you mouse over the text in the search results, and therefore what is added to your query when clicked on. Let's see what happens to the phrase ip=10.20.30.40 with each setting:

• inner highlights individual words between punctuation. Highlighted items would be ip, 10, 20, 30, and 40.

• outer...

Using fields to search

When we explored the GUI in Chapter 1, The Splunk Interface, you probably noticed fields everywhere. Fields appear in the field picker on the left and under every event. Where fields actually come from is transparent to the user, who simply searches for key=value. We will discuss adding new fields in Chapter 3, Tables, Charts, and Fields, and in Chapter 10, Configuring Splunk.

Using the field picker

The field picker gives us easy access to the fields currently defined for the results of our query. Clicking on any field presents us with details about that field in our current search results.

As we go through the following items in this widget, we see a wealth of information right away:

• Appears in X% of results is a good indication of whether we are getting the results we think we're getting. If every event in your results should contain this field, and this is not 100 percent, either your search can be made more specific or a field definition needs to be modified.

• Show...

Using wildcards efficiently

Though the index is based on words, it is possible to use wildcards when needed, although some care must be taken.

Only trailing wildcards are efficient

Stated simply, bob* will find events containing Bobby efficiently, but *by or *ob* will not. The latter cases will scan all events in the time frame specified.

Wildcards are tested last

Wildcards are tested after all other terms. Given the search: authclass *ob* hello world, all other terms besides *ob* will be searched first. The more you can limit the results using full words and fields, the better your search will perform.

Supplementing wildcards in fields

Given the following events, a search for world would return both events:

2012-02-07T01:04:31.102-0600 INFO AuthClass Hello world. [user=Bobby, ip=1.2.3.3]
2012-02-07T01:23:34.204-0600 INFO BarClass Goodbye. [user=Bobby, ip=1.2.3.3, message="Out of this world"]

What if you only wanted the second event, but all you know is that the event contains world somewhere in...

All about time

Time is an important and confusing topic in Splunk. If you want to skip this section, absorb one concept—time must be parsed properly on the way into the index as it cannot be changed later without indexing the raw data again.

How Splunk parses time

Given the date 11-03-04, how would you interpret this date? Your answer probably depends on where you live. In the United States, you would probably read this as November 3, 2004. In Europe, you would probably read this as March 11, 2004. It would also be reasonable to read this as March 4, 2011.

Luckily, most dates are not this ambiguous, and Splunk makes a good effort. It is absolutely worth the trouble to give Splunk a little help by configuring the time format. We'll discuss the relevant configurations in Chapter 10, Configuring Splunk.

How Splunk stores time

Once the date is parsed, the date stored in Splunk is always stored as GMT epoch. Epoch time is the number of seconds since January 1, 1970, the birthday of Unix. By storing...

Making searches faster

We have talked about using the index to make searches faster. When starting a new investigation, following a few steps will help you get results faster:

1. Set the time to the minimum time that you believe will be required to locate relevant events. For a chatty log, this may be as little as a minute. If you don't know when the events occurred, you might search a larger time frame and then zoom in by clicking on the timeline while the search is running.

2. Specify the index if you have multiple indexes. It's good to get into the habit of starting your queries with the index name, for example, index=myapplicationindex error bob.

3. Specify other fields that are relevant. The most common fields to specify are sourcetype and host, for example, index=myapplicationindex sourcetype="impl_splunk_gen" error bob.

   Note

   If you find yourself specifying the field source on a regular basis, you could probably benefit from defining more source types. Avoid using the sourcetype field to capture...

Sharing results with others

It is often convenient to share a specific set of results with another user. You could always export the results to a CSV file and share it, but this is cumbersome. Instead, to use a URL for sharing, start by choosing Save & share results… from the Save menu.

This opens the Save and Share Results panel.

The URL under Link to the results can be copied and sent to other users. A user visiting this URL will see exactly the same results you did, assuming the job has not expired.

The results are also available in the Jobs window. Clicking on the Jobs link in the top bar opens the Jobs window.

The App menu, Owner menu, Status menu, and search bar let you filter what jobs are displayed.

The table has the following columns:

• Dispatched at is the time at which the search started.

• Owner is the user that started the job. Sometimes jobs will appear with system as the user if the saved search is configured in an application but not owned by a particular user.

• Application specifies...

Saving searches for reuse

Let's build a query, save it, and make an alert out of it.

First, let's find errors that affect mary, one of our most important users. This can simply be the query mary error. Looking at some sample log messages that match this query, we see that some of these events probably don't matter (the dates have been removed to shorten the lines).

  ERROR LogoutClass error, ERROR, Error! [user=mary, ip=3.2.4.5]
  WARN AuthClass error, ERROR, Error! [user=mary, ip=1.2.3.3]
  ERROR BarCLass Hello world. [user=mary, ip=4.3.2.1]
  WARN LogoutClass error, ERROR, Error! [user=mary, ip=1.2.3.4]
  DEBUG FooClass error, ERROR, Error! [user=mary, ip=3.2.4.5]
  ERROR AuthClass Nothing happened. This is worthless. Don't log this.[user=mary, ip=1.2.3.3]

We can probably skip the DEBUG messages; the LogoutClass messages look harmless; and the last message actually says that it's worthless.

mary error NOT debug NOT worthless NOT logoutclass limits the results to:

  WARN AuthClass error...

Creating alerts from searches

Any saved search can also be run on a schedule. One use for scheduled searches is firing alerts. To get started, choose Alert… from the Create menu.

A wizard interface is presented, covering three steps.

Schedule

The Schedule step provides the following options:

• Trigger in real-time whenever a result matches: This option will leave a real-time search running all the time and will immediately fire an alert whenever an event is seen.

  

  This option will create an alert every time an event that matches your search occurs. There is an important throttling option in the next step.

• Run on a schedule once every…: New options now appear below the menu.

  

  • Schedule: You can choose to either run your search on a set schedule or run your alert according to a cron schedule. Keep in mind that the time frame selected in the time picker will be used each time the query runs—you probably don't want to run a query looking at 24 hours of data every minute.

  • Trigger if lets you choose when...

Summary

In this chapter, we covered searching in Splunk and doing a few useful things with those search results. There are lots of little tricks that we will touch upon as we go forward.

In the next chapter, we will start using fields for more than searches; we'll build tables and graphs, and then, we'll learn how to make our own fields.