Working with Apps in Splunk

(For more resources related to this topic, see here.)

Defining an app

In the strictest sense, an app is a directory of configurations and, sometimes, code. The directories and files inside have a particular naming convention and structure. All configurations are in plain text, and can be edited using your choice of text editor.

Apps generally serve one or more of the following purposes:

  1. A container for searches, dashboards, and related configurations: This is what most users will do with apps. This is not only useful for logical grouping, but also for limiting what configurations are applied and at what time. This kind of app usually does not affect other apps.

  2. Providing extra functionality: Many objects can be provided in an app for use by other apps. These include field extractions, lookups, external commands, saved searches, workflow actions, and even dashboards. These apps often have no user interface at all; instead they add functionality to other apps.

  3. Configuring a Splunk installation for a specific purpose: In a distributed deployment, there are several different purposes that are served by the multiple installations of Splunk. The behavior of each installation is controlled by its configuration, and it is convenient to wrap those configurations into one or more apps. These apps completely change the behavior of a particular installation.

Included apps

Without apps, Splunk has no user interface, rendering it essentially useless. Luckily, Splunk comes with a few apps to get us started. Let's look at a few of these apps:

  • gettingstarted: This app provides the help screens that you can access from the launcher. There are no searches, only a single dashboard that simply includes an HTML page.

  • search: This is the app where users spend most of their time. It contains the main search dashboard that can be used from any app, external search commands that can be used from any app, admin dashboards, custom navigation, custom css, a custom app icon, a custom app logo, and many other useful elements.

  • splunk_datapreview: This app provides the data preview functionality in the admin interface. It is built entirely using JavaScript and custom REST endpoints.

  • SplunkDeploymentMonitor: This app provides searches and dashboards to help you keep track of your data usage and the health of your Splunk deployment. It also defines indexes, saved searches, and summary indexes. It is a good source for more advanced search examples.

  • SplunkForwarder and SplunkLightForwarder: These apps, which are disabled by default, simply disable portions of a Splunk installation so that the installation is lighter in weight.

If you never create or install another app, and instead simply create saved searches and dashboards in the app search, you can still be quite successful with Splunk. Installing and creating more apps, however, allows you to take advantage of others' work, organize your own work, and ultimately share your work with others.

Installing apps

Apps can either be installed from Splunkbase or uploaded through the admin interface. To get started, let's navigate to Manager | Apps, or choose Manage apps... from the App menu as shown in the following screenshot:

Installing apps from Splunkbase

If your Splunk server has direct access to the Internet, you can install apps from Splunkbase with just a few clicks. Navigate to Manager | Apps and click on Find more apps online. The most popular apps will be listed as follows:

Let's install a pair of apps and have a little fun. First, install Geo Location Lookup Script (powered by MAXMIND) by clicking on the Install free button. You will be prompted for your login. This is the same login that you created when you downloaded Splunk. If you don't have an account, you will need to create one.

Next, install the Google Maps app. This app was built by a Splunk customer and contributed back to the Splunk community. This app will prompt you to restart Splunk.

Once you have restarted and logged back in, check the App menu.

Google Maps is now visible, but where is Geo Location Lookup Script? Remember that not all apps have dashboards; nor do they necessarily have any visible components at all.

Using Geo Location Lookup Script

Geo Location Lookup Script provides a lookup script to provide geolocation information for IP addresses. Looking at the documentation, we see this example:

eventtype=firewall_event | lookup geoip clientip as src_ip

You can find the documentation for any Splunkbase app by searching for it at, or by clicking on Read more next to any installed app by navigating to Manager | Apps | Browse more apps.

Let's read through the arguments of the lookup command:

  • geoip: This is the name of the lookup provided by Geo Location Lookup Script.

    You can see the available lookups by going to Manager | Lookups | Lookup definitions.

  • clientip: This is the name of the field in the lookup that we are matching against.

  • as src_ip: This says to use the value of src_ip to populate the field before it; in this case, clientip. I personally find this wording confusing. In my mind, I read this as "using" instead of "as".

Included in the ImplementingSplunkDataGenerator app (available at is a sourcetype instance named impl_splunk_ips, which looks like this:

2012-05-26T18:23:44 ip=

The IP addresses in this fictitious log are from one of my websites. Let's see some information about these addresses:

sourcetype="impl_splunk_ips" | lookup geoip clientip AS ip | top client_country

This gives us a table similar to the one shown in the following screenshot:

That's interesting. I wonder who is visiting my site from Slovenia!

Using Google Maps

Now let's do a similar search in the Google Maps app. Choose Google Maps from the App menu. The interface looks like the standard search interface, but with a map instead of an event listing. Let's try this remarkably similar (but not identical) query using a lookup provided in the Google Maps app:

sourcetype="impl_splunk_ips" | lookup geo ip

The map generated looks like this:

Unsurprisingly, most of the traffic to this little site came from my house in Austin, Texas.

Installing apps from a file

It is not uncommon for Splunk servers to not have access to the Internet, particularly in a datacenter. In this case, follow these steps:

  1. Download the app from The file will have a .spl or .tgz extension.

  2. Navigate to Manager | Apps.

  3. Click on Install app from file.

  4. Upload the downloaded file using the form provided.

  5. Restart if the app requires it.

  6. Configure the app if required.

That's it. Some apps have a configuration form. If this is the case, you will see a Set up link next to the app when you go to Manager | Apps. If something goes wrong, contact the author of the app.

If you have a distributed environment, in most cases the app only needs to be installed on your search head. The components that your indexers need will be distributed automatically by the search head. Check the documentation for the app.

Building your first app

For our first app, we will use one of the templates provided with Splunk. To get started, navigate to Manager | Apps and then click on Create app. The following page will open:

Set the fields as follows:

  • Set Name to Implementing Splunk App One. This name will be visible on the home screen, in the App menu, and in the app banner in the upper left of the window.

  • Set Folder name to is_app_one. This value will be the name of the app directory on the filesystem, so you should limit your name to letters, numbers, and underscores.

  • Set Visible to Yes. If your app simply provides resources for other apps to use, there may be no reason for it to be visible.

  • Set Template to barebones. The barebones template contains sample navigation and the minimal configuration required by an app. The sample_ app template contains many example dashboards and configurations.

After clicking on Save, we can now visit our app by going to Manager | Apps, in the App menu, and in the Home app.

Now that we have our app, we can create searches and dashboards, and maintain them in our app. The simplest way to ensure that your objects end up in your app is to verify that the app banner is correct before creating objects or before entering the Splunk Manager. Our app banner looks like this:

Create a dashboard called Errors using the following searches

error sourcetype="impl_splunk_gen" | timechart count by user error sourcetype="impl_splunk_gen" | top user error sourcetype="impl_splunk_gen" | top logger

This produces the following result:

The searches appear under Searches & Reports, and our new dashboard appears in the navigation menu under Views:

Editing navigation

Navigation is controlled by an XML file that can be accessed by going to Manager | User interface | Navigation menus.

There can only be one active navigation file per app, and it is always called default. After clicking on the name, we see the XML provided by the barebones template:

<nav> <view name="flashtimeline" default='true' /> <collection label="Dashboards"> <view source="unclassified" match="dashboard"/> <divider /> </collection> <collection label="Views"> <view source="unclassified" /> <divider /> </collection> <collection label="Searches & Reports"> <collection label="Reports"> <saved source="unclassified" match="report" /> </collection> <divider /> <saved source="unclassified" /> </collection> </nav>

The structure of the XML is essentially the following:

nav view saved collection view a href saved divider collection

The logic of navigation is probably best absorbed by simply editing it and seeing what happens. You should keep a backup, as this XML is somewhat fragile and Splunk does not provide any kind of version control. Here are some general details about nav:

  • Children of nav appear in the navigation bar.

  • collection: Children of collection tags appear in a menu or submenu. If the child tags do not produce any results, the menu will not appear. The divider tag always produces a result, so it can be used to ensure that a menu appears.

  • view: This tag represents a dashboard, with the following attributes:

    • name is the name of the dashboard filename, without .xml.

    • The first view element with the attribute default='true' will load automatically when the app is selected.

    • The label of each view is based on the contents of the label tag in the dashboard XML, not the name of the dashboard filename.

    • match="dashboard" selects all dashboards whose filename contains dashboard. If you want to group dashboards, you may want to follow a naming convention to make grouping more predictable.

    • source="unclassified" essentially means "all views that have not been previously associated to a menu". In other words, this will match dashboards that were not explicitly referenced by name or matched using the match attribute or a different view tag.

  • a href: You can include standard HTML links of the form <a href="http://a.b/c".

    The link is untouched and passed along as written.

  • saved: This tag represents a saved search, with the following attributes:

    • name is equal to the name of a saved search.

    • match="report" selects all saved searches that have report in their names.

    • source="unclassified" essentially means "all searches that have not yet been previously associated to a menu". In other words, this will match searches that were not explicitly referenced by name or matched using the match attribute or a different saved tag.

Let's customize our navigation. We'll make a few changes like these:

  • Create an entry specifically for our errors dashboard

  • Add default='true' so that this dashboard loads by default

  • Simplify the Views and Searches collections

These changes are reflected in the following code:

<nav> <view name="errors" default='true' /> <view name="flashtimeline" /> <collection label="Views"> <view source="unclassified" /> </collection> <collection label="Searches"> <saved source="unclassified" /> </collection> </nav>

Our navigation now looks like this screenshot:

With this navigation in place, all new dashboards will appear under Views, and all new saved searches will appear under Searches.

Notice that Advanced Charting and Google Maps appear under Views. Neither of these dashboards are part of our app, but are visible because of the permissions in their respective apps. We will discuss permissions in more detail in the Object permissions section.

Customizing the appearance of your app

It is helpful to further customize the appearance of your application, if for no other reason than to make it more obvious which app is currently active.

Customizing the launcher icon

The launcher icon is seen both in the Home app and in Splunkbase, if you decide to share your app. The icon is a 36 x 36 PNG file named appIcon.png. I have created a simple icon for our sample app (please don't judge my art skills):

To use the icon follow these steps:

  1. Navigate to Manager | Apps.

  2. Click on Edit properties next to our app, Implementing Splunk App One.

  3. Click on Upload asset and select the file.

  4. Click on Save.

Our icon will now appear on the launcher screen, like in the following screenshot:

Using custom CSS

The look of the Splunk application is controlled via CSS. One common element to change is the application icon in the application bar. Follow these steps to do just that:

  1. First, create a file named application.css. This file will be loaded on every dashboard of the application containing it. The CSS is listed later in this section.

    As of Splunk Version 4.3.2, the first time application.css is added to an app of Version 4.3.2, a restart is required before the file is served to the users. Subsequent updates do not require a restart.

  2. Next, create a file named appLogo.png. This file can be called anything, as we will reference it explicitly in our CSS file. Borrowing CSS from the search app, we will make our file 155 x 43 pixels:

  3. For each file, follow the same steps as for uploading the launcher icon:

    1. Navigate to Manager | Apps.

    2. Click on Edit properties next to our app, Implementing Splunk App One.

    3. Click on Upload asset and select the file.

    4. Click on Save.

Our CSS references a few classes in the application header bar:

.appHeaderWrapper h1 { display: none; } .appLogo { height: 43px; width: 155px; padding-right: 5px; float: left; background: url{appLogo.png} no-repeat 0 0; } .appHeaderWrapper { background: #612f00; }

Let's step through these classes:

  • .appHeaderWrapper h1: By default, the name of the app appears as text in the upper-left corner. This definition hides that text.

  • .appLogo: This sets the background of the upper-left block to our custom file. The height and width should match the dimensions of our logo.

  • .appHeaderWrapper: This sets the background color of the top bar.

With everything in place, our top bar now looks like this:

Using custom HTML

In some apps, you will see static HTML blocks. This can be accomplished using both simple and complex dashboards.

Custom HTML in a simple dashboard

In a simple dashboard, you can simply insert anelement inside a <row> element, and include static HTML inline. For example, after uploading an image named graph.png, the following block can be added to any dashboard:

<row> <html> <table> <tr> <td><img src = " /static/app/is_app_one/graph.png" /></td> <td> <p>Lorem ipsum ...</p> <p>Nulla ut congue ...</p> <p>Etiam pharetra ...</p> </td> </tr> </table> </html> </row>

The XML would render this panel:

This approach has the advantage that no other files are needed. The disadvantage, however, is that you cannot build the HTML document in an external program and upload it untouched.

You could also reference custom CSS using this method by adding classes to application.css and then referencing those classes in your HTML block.

Using ServerSideInclude in a complex dashboard

You can also develop static pages as HTML documents, referencing other files in the same directory. Let's build a slightly more complicated page using graph.png, but also a style from application.css as follows:

  1. Place graph.png and application.css into a directory.

  2. Create a new HTML file. Let's name it intro.html.

  3. Add any styles for your page to application.css.

  4. Upload the new HTML file and modified CSS file.

  5. Create the dashboard referencing the HTML file.

Starting with the HTML from our previous example, let's make it a complete document: move the image to a CSS style and add a class to our text, like this:

<html> <head> <link rel="stylesheet" type="text/css" href="application.css" /> </head> <body> <table> <tr> <td class="graph_image"></td> <td> <p class="lorem">Lorem ipsum ...</p> <p class="lorem">Nulla ut congue ...</p> <p class="lorem">Etiam pharetra ...</p> </td> </tr> </table> </body> </html>

Maintaining the classes for the navigation bar, add our page classes to application. css, like this:

.appHeaderWrapper h1 { display: none; } .appLogo { height: 43px; width: 155px; padding-right: 5px; float: left; background: url{appLogo.png} no-repeat 0 0; } .appHeaderWrapper { background: #612f00; } .lorem { font-style:italic; background: #CCCCCC; padding: 5px; } .graph_image { height: 306px; width: 235px; background: url{graph.png} no-repeat 0 0; }

We can now open this file in a browser. Clipped for brevity, the page looks like this:

To include this external HTML document, we have to use advanced XML.

First, build a minimal dashboard like this:

<view template="dashboard.html"> <label>Included</label> <!-- chrome here --> <module name="ServerSideInclude" layoutPanel="panel_row1_col1"> <param name="src">intro.html</param> </module> </view>

All "simple" XML dashboards are converted to "advanced" XML behind the scenes. We will take advantage of this later.

Now upload our files as we did before under the Customizing the launcher icon section. The page should render nearly identically as the file did in the browser, with the addition of the border around the panel:

A few things to note from this overly simplified example are as follows:

  1. Your CSS classes may end up merging with styles included by Splunk in unexpected ways. Using the developer tools in any modern browser will help greatly.

  2. The navigation and dashboard title were excluded for brevity. They would normally go where we see <!-- chrome here -->. This is interesting because there are cases where you would want to exclude the navigation; something that cannot be done with simple XML.

  3. The static files, such as application.css, can be edited directly on the filesystem, and the changes will be seen immediately. This is not true of the dashboard XML file. We will cover these locations later in the App directory structure section.

Object permissions

Almost all objects in Splunk have permissions associated with them. The permissions essentially have the following three options:

  • Private: Only the user that created the search can see or use the object, and only in the app where it was created

  • App: All users that have permission to read an object may use that object in the context of the app that contains that object

  • Global: All users that have permission to read an object may use that object in any app

How permissions affect navigation

To see a visible instance of permissions in action, let's look at our navigation. In our application, Implementing Splunk App One, our navigation looks like this:

If you recall the navigation XML we built before, this menu is controlled by the following XML:

<collection label="Views"> <view source="unclassified" /> </collection>

There is no mention of any of these dashboards. Here is where they are coming from:

  • Advanced Charting is inherited from the Search app. Its permissions are set to Global.

  • Included is from this app. Its permissions are set to App.

  • Google Maps is inherited from the Google Maps app. Its permissions are set to Global.

If the permissions of a dashboard or search are set to Private, a green dot appears next to the name in the navigation.

Dashboards or searches shared from other apps can also be referenced by name. For example, most apps, including ours, will include a link to flashtimeline, which appears as Search, the label in that dashboard's XML:

<view name="flashtimeline" />

This allows us to use this dashboard in the context of our app so that all of the other objects that are scoped solely to our app will be available.

How permissions affect other objects

Almost everything you create in Splunk has permissions. To see all objects, navigate to Manager | All configurations.

Everything with the value system in the App column ships with Splunk. These items live in $SPLUNK_HOME/etc/system.

When you create new objects and configurations, it is important to share all related objects. It is important that all three parts of the lookup definition are shared appropriately, or users will be presented with error messages.

Correcting permission problems

If you see errors about permissions, it is more than likely that some object still has Sharing set to Private, or is shared at the App level but needs to be Global. Follow these steps to find the object:

  1. Navigate to Manager | All configurations.

  2. Change App context to All.

  3. Sort by using the Sharing status. Click twice so that Private objects come to the top.

  4. If there are too many items to look through, filter the list by adding terms to the search field in the upper-right corner, or changing the App context value.

  5. Fix the permissions appropriately. In most cases, the permissions you want will look like this:

You should choose All apps with care. For instance, when building a lookup, it is common to share the lookup table file and lookup definition across all apps. This allows the lookup to be used in searches by other apps. It is less common to share the Automatic lookup, as this can affect performance in other apps in unforeseen ways.

App directory structure

If you do much beyond building searches and dashboards, sooner or later you will need to edit files in the filesystem directly. All apps live in $SPLUNK_HOME/etc/ apps/. On Unix systems, the default installation directory is /opt/splunk. On Windows, the default installation directory is c:\Program Files\Splunk. This is the value that $SPLUNK_HOME will inherit on startup.

Stepping through the most common directories, we have:

  • appserver: This directory contains files that are served by the Splunk web app. The files that we uploaded in earlier sections of this article are stored in appserver/static.

  • bin: This is where command scripts belong. These scripts are then referenced in commands.conf. This is also a common location for scripted inputs to live, though they can live anywhere.

  • default and local: These two directories contain the vast majority of the configurations that make up an app.. Here is a brief look:

    • Newly created, unshared objects live in $SPLUNK_HOME/etc/users/ USERNAME/APPNAME/local.

    • Once an object is shared at the App or Global level, the object is moved to $SPLUNK_HOME/etc/APPNAME/local.

    • Files in local take precedence over its equivalent value in default.

    • Dashboards live in (defaultlocal)/data/ui/views.|

    • Navigations lives in (defaultlocal)/data/ui/nav.|

    • When editing files by hand, my general rule of thumb is to place configurations in local unless the app will be redistributed. We'll discuss this in more detail in the Adding your app to Splunkbase section.

  • lookups: Lookup files belong in this directory. They are then referenced in (defaultlocal)/transforms.conf.|

  • metadata: The files default.meta and local.meta in this directory tell Splunk how configurations in this app should be shared. It is generally much easier to edit these settings through the Manager interface.

Let's look at the contents of our is_app_one app, which we created earlier:

appserver/static/appIcon.png appserver/static/application.css appserver/static/appLogo.png appserver/static/graph.png appserver/static/intro.html bin/README default/app.conf default/data/ui/nav/default.xml default/data/ui/views/README local/app.conf local/data/ui/nav/default.xml local/data/ui/views/errors.xml local/data/ui/views/included.xml local/savedsearches.conf local/viewstates.conf metadata/default.meta metadata/local.meta

The file metadata/default.meta, and all files in default/, were provided in the template app. We created all of the other files. With the exception of the png files, all files are plain text.

Adding your app to Splunkbase

Splunkbase ( is a wonderful community-supported site that Splunk put together for users and Splunk employees alike to share Splunk apps. The apps on Splunkbase are a mix of fully realized apps, add-ons of various sorts, and just example code. Splunk has good documentation for sharing apps at the following URL: ShareYourWork

Preparing your app

Before we upload our app, we need to make sure all of our objects are shared properly, move our files to default, and configure app.conf.

Confirming sharing settings

To see sharing settings for all our objects, navigate to Manager | All configurations and set the App context option:

In the case of a self-contained app like ours, all objects should probably be set to App under Sharing. If you are building an app to share lookups or commands, the value should be Global.

Cleaning up our directories

When you upload an app, you should move everything out of local and into default. This is important because all changes a user makes will be stored in local. When your app is upgraded, all files in the app will be replaced, and the user's changes will be lost. The following Unix commands illustrate what needs to be done:

  1. First, let's copy our app to another location, perhaps /tmp:

    cp -r $SPLUNK_HOME/etc/apps/is_app_one /tmp/

  2. Next, let's move everything from local to default. In the case of .xml files, we can simply move the files; but .conf files are a little more complicated, and we need to merge them manually. The following code does this:

    cd /tmp/is_app_one mv local/data/ui/nav/*.xml default/data/ui/nav/ mv local/data/ui/views/*.xml default/data/ui/views/ #move conf files, but don't replace conf files in default mv -n local/*conf default/

  3. Now we need to merge any .conf files that remain in local. The only configuration we have left is app.conf;





    is_configured = 0



    is_visible = 1

    label = Implementing Splunk

    App One


    check_for_updates = 1


    author =

    description =

    version = 1.0

    Configuration merging is additive, with any values from local added to the values in default. In this case, the merged configuration would be as follows:

    [install] is_configured = 0 [ui] is_visible = 1 label = Implementing Splunk App One [launcher] author = description = version = 1.0 [package] check_for_updates = 1

  4. Place this merged configuration in default/app.conf and delete local/ app.conf.

Packaging your app

To package an app, we need to be sure that there are a few values in default/app. conf, and only then build the archive.

First, edit default/app.conf like this:

[install] is_configured = 0 build = 1 [ui] is_visible = 1 label = Implementing Splunk App One [launcher] author = My name description = My great app! version = 1.0 [package] check_for_updates = 1 id = is_app_one

build is used in all URLs, so it should be incremented to defeat browser caching. id should be a unique value in Splunkbase—you will be alerted if the value is not unique.

Next, we need to build a tar file compressed with gzip. With a modern version of tar, the command is simply the following:

cd /tmp tar -czvf is_app_one.tgz is_app_one #optionally rename as spl mv is_app_one.tgz is_app_one.spl

The Splunk documentation ( latest/AdvancedDev/PackageApp) covers this extensively, including Mac and Windows procedures.

Uploading your app

Now that we have our archive, all we have to do is send it up to Splunkbase. First, click on the upload an app button.

Then fill out the form shown in the following screenshot:

More than likely your app will not be compatible with Splunk 3.x, so uncheck the 3.x checkbox.

Once Splunk personnel approve your app, it will appear in Splunkbase, ready for others to download.


In this article, we covered installing, building, customizing, and sharing apps. Apps are a loose concept in Splunk, with many different purposes served by a simple directory of files. Hopefully we have covered the basics well enough for you to get started on your own great apps.

Resources for Article :

Further resources on this subject:

You've been reading an excerpt of:

Implementing Splunk: Big Data Reporting and Development for Operational Intelligence

Explore Title