Chapter 12. Extending Splunk

While the core of Splunk is closed, there are a number of places where you can use scripts or external code to extend the default behaviors. In this chapter, we will write a number of examples, covering most of the places where external code can be added. Most code samples are written in Python, so if you are not familiar with Python, a reference may be useful.

We will cover:

• Writing scripts to create events

• Using Splunk from the command line

• Calling Splunk via REST

• Writing custom search commands

• Writing event type renderers

• Writing custom search action scripts

The examples used in this chapter are included in the app ImplementingSplunkExtendingExamples, which can be downloaded from the support page of the Packt Publishing website (www.packtpub.com/support).

Writing a scripted input to gather data

Scripted inputs allow you to run some piece of code on a scheduled basis, and capture the output as if it were simply being written to a file. It does not matter what language the script is written in, or where it lives, as long it is executable. We touched on this topic in the Using scripts to gather data section in Chapter 11, Advanced Deployments. Let's write a few more examples.

Capturing script output with no date

One common problem with script output is the lack of a predictable date or date format. In this situation, the easiest thing to do is to tell Splunk to not try to parse a date at all, and instead use the current date instead. Let's make a script that lists open network connections:

from subprocess import Popen
from subprocess import PIPE
from collections import defaultdict
import re


def add_to_key(fieldname, fields):
    return " " + fieldname + "+" + fields[fieldname]

output = Popen("netstat -n -p tcp", stdout=PIPE, 
             ...

Using Splunk from the command line

Almost everything that can be done via the web interface can also be accomplished via the command line. For an overview, see the output of /opt/splunk/bin/splunk help. For help on a specific command, use /opt/splunk/bin/splunk help [commandname].

The most common action to perform on the command line is search. For example, have a look at the following code:

$ /opt/splunk/bin/splunk search 'foo'
2012-08-25T20:17:54 user=user2 GET /foo?q=7148356 uid=MzA4MTc5OA
2012-08-25T20:17:54 user=user2 GET /foo?q=7148356 uid=MzA4MTc5OA
2012-08-25T20:17:54 user=user2 GET /foo?q=7148356 uid=MzA4MTc5OA
...

Things to note:

• By default, searches are performed over All time. Protect yourself by including earliest=-1d or an appropriate time range in your query.

• By default, Splunk will only output 100 lines of results. If you need more, use the -maxout flag.

• Search requires authentication, so the user will be asked to authenticate unless -auth is included as an argument.

Most use...

Querying Splunk via REST

Splunk provides an extensive HTTP REST interface, which allows searching, adding data, adding inputs, managing users, and more. Documentation and SDKs are provided by Splunk at http://dev.splunk.com/.

To get an idea of how this REST interaction happens, let's step through a sample conversation to run a query and retrieve the results. The steps are essentially as follows:

1. Start the query (POST).

2. Poll for status (GET).

3. Retrieve results (GET).

We will use the command line program cURL to illustrate these steps. The SDKs make this interaction much simpler.

To start a query, the command is as follows:

curl -u user:pass -k https://yourserver:8089/services/search/jobs
  -d"search=search query"

This essentially says to POST search=search query. If you are familiar with HTTP, you might notice that this is a standard POST from an HTML form.

To run the query earliest=-1h index="_internal" warn | stats count by host, we need to URL encode the query. The command then is as follows:

...

Writing commands

To augment the built-in commands, Splunk provides the ability to write commands in Python and Perl. You can write the commands to modify events, replace events, or even dynamically produce events.

When not to write a command

While external commands can be very useful, if the number of events to be processed is large, or if performance is a concern, it should be considered a last resort. You should make every effort to accomplish the task at hand using the search language built into Splunk, or other built-in features. For instance, if you need:

• Regular expressions—learn to use rex, regex, and extracted fields

• To calculate a new field, or modify an existing field—look into eval (search for splunk eval functions with your favorite search engine)

• To augment your results with external data—learn to use lookups , which can also be a script, if need be

• To read external data that changes periodically—consider using inputcsv

The performance issues introduced by external commands come from...

Writing a scripted lookup to enrich data

We covered CSV lookups fairly extensively in Chapter 6, Extending Search, then touched on them again in Chapter 9, Summary Indexes and CSV Files and Chapter 10, Configuring Splunk. The capabilities built into Splunk are usually sufficient, but sometimes it is necessary to use an external data source or dynamic logic to calculate values. Scripted lookups have the following advantages over commands or CSV lookups:

• Scripted lookups are only run once per unique lookup value, as opposed to a command, which would run the command for every event

• The memory requirement of a CSV lookup increases with the size of the CSV file

• Rapidly changing values can be left in an external system and queried using the scripted lookup instead of being exported frequently

In the Using a lookup with wildcards section in Chapter 9, Summary Indexes and CSV Files, we essentially created a case statement through configuration. Let's implement that use case as a script, just to show...

Writing an event renderer

Event renderers give you the ability to make a specific template for a specific event type. To read more about creating event types, see Chapter 6, Extending Search.

Event renderers use mako templates (http://www.makotemplates.org/). An event renderer is comprised of the following:

• A template stored at $SPLUNK_HOME/etc/apps/[yourapp]/appserver/event_renderers/[template].html

• A configuration entry in event_renderers.conf

• An optional event type definition in eventtypes.conf

• Optional CSS classes in application.css

Let's create a few small examples. All the files referenced are included in $SPLUNK_HOME/etc/apps/ImplementingSplunkExtendingExamples. These examples are not shared outside this app, so to see them in action, you will need to search from inside this app. Do this by pointing your browser at http://[yourserver]/app/ ImplementingSplunkExtendingExamples/flashtimeline.

Using specific fields

If you know the names of the fields you want to display in your output, your...

Writing a scripted alert action to process results

Another option for interfacing with an external system is to run a custom Alert action using the results of a saved search. Splunk provides a simple example in $SPLUNK_HOME/bin/scripts/echo.sh. Let's try it out and see what we get, using the following steps:

1. Create a saved search. For this test, do something cheap, such as the following:

   index=_internal | head 100 | stats count by sourcetype

2. Schedule the search to run at some point in the future. I set it to run every five minutes, just for this test.

3. Enable Run a script and type in echo.sh.

   

The script places the output into $SPLUNK_HOME/bin/scripts/echo_output.txt. In my case, the output is as follows:

'/opt/splunk/bin/scripts/echo.sh' '4' 'index=_internal | head 100 | stats count by sourcetype' 'index=_internal | head 100 | stats count by sourcetype' 'testingAction' 'Saved Search [testingAction] always(4)' 'http://vlbmba.local:8000/app/search/@go?sid=scheduler__admin__search__testingAction_at_1352667600_2efa1666cc496da4...

Summary

As we have seen in this chapter, there are a number of ways in which Splunk can be extended to input, manipulate, and output events. The search engine at the heart of Splunk is truly just the beginning. With a little creativity, Splunk can be used to extend existing systems, both as a data source and as a way to trigger actions.