Reporting, Alerts, and Search Optimization

Finding the data you need in Splunk is relatively easy, as you have seen in previous chapters. Doing the same thing repeatedly for different datasets, however, requires that you employ techniques that make data retrieval faster, easier, and more controlled with reusable configurations. In Chapter 2, Bringing in Data, you were shown how to use data fields and make field extractions. In Chapter 6, Data Models and Pivot, you learned how to create data models. You will continue that journey in this chapter by learning how to classify your data using Event Types, enrich your data using Lookups, and normalize your data using Tags.

Once you have these essentials in place, you will be able to more easily create reports, alerts, and dashboards, and capture analytical value from machine data quickly.

In this chapter, we will cover a wide range...

Data classification with Event Types

When working with Splunk daily, you will find many of the tasks and searches you run are repeated on a periodic basis. As shown earlier, storing field extraction logic in a single place allows it to be reused in the future. Another way to make things easier and also shorten searches is to create Event Types. Event Types are not the same as events; an event is just a single instance of data. An Event Type is a grouping or classification of events meeting the same search criteria.

If you took a break between chapters, you will probably want to open up Splunk again. Then, execute a search command:

1. Log in to the Splunk portal
2. Click on your Destinations app
3. Type this search in the search bar:

SPL> index=main http_uri=/booking/confirmation http_status_code=200 

This search will return events representing successful booking confirmations. In...

Data normalization with Tags

Tags in Splunk are useful for grouping events with related field values. Unlike Event Types, which are based on specified search commands, Tags are created and mapped to specific field-value combinations. Multiple Tags can be assigned to the same field-value combination.

A common scenario of using Tags is for classifying IP addresses. In the Eventgen logs, three IP addresses are automatically generated. We will create Tags against these IP addresses to allow us to classify them:

IP address	Tags
10.2.1.33	main, patched, and east
10.2.1.34	main, patched, and west
10.2.1.35	backup and east

We are going to group IP addresses by purpose, patch status, and geolocation in the server farm of three servers represented in our Eventgen data. We will achieve this using Tags, as shown in the following steps:

1. Begin by using the following search command...

Data enrichment with Lookups

Occasionally you will require pieces of data to be rendered in a more readable manner. A common example to go through using our Eventgen data is with HTTP status. Computer engineers are often familiar with HTTP status codes as three-digit numbers. Business analysts or more casual users may not know the meaning of these codes and require a text-based description to comprehend the search results. In Splunk, you can enrich event data using Lookups, which can pair numbers or acronyms with more understandable text descriptions found in a separate file.

A lookup table is a mapping of keys and values Splunk can search, allowing for the displaying of more meaningful information at search time. Having the Lookup execute at search run time also optimizes the need to index verbose descriptions that consume additional index space. This is best understood through...

Creating and scheduling reports

In this chapter, you have learned three very important things: classifying raw events using Event Types, classifying data using Tags, and enriching data using Lookups. These, in addition to good SPL, constitute essential elements you need to use Splunk in an efficient manner.

Splunk reports are saved searches which can be shared to others or used as a dashboard panel. Reports can be scheduled periodically and perform an action upon completion, such as sending an email with the report results.

Reports can be configured to display search results in a statistical table, as well as visualization charts. A report is created using the search command line or through a Pivot. Here we will create a report using the search command line:

1. In the Destinations app's search page, run the following search:

SPL> eventtype=bad_logins | top client_ip 

The...

Creating alerts

Alerts are crucial in IT and security operations. They provide proactive awareness of the state of the systems to those persons who monitor and control them. Alerts enable you to act fast when an issue has been detected, as opposed to waiting for a user to run a report and find the issue, which may or may not happen. In today's world, every minute someone has breached your network is costly and potentially devastating.

However, alerts are only good if they are controlled and if they provide enough actionable information. They should not be created on low-priority items or triggered too often to the point they lose relevance.

Tip from the Fez: Out-of-the box functionality for alerts is most commonly driven to email. Users may also want to explore the use of text messages. When Splunk doesn't provide something out of the box, there is a good chance the...

Search and Report acceleration

In Chapter 6, Data Models and Pivot, you learned how to accelerate a data model to speed up retrieval of data. The same principle applies to saved searches or reports:

1. Click on the Reports link in the navigation menu of the Destinations app
2. Click on the Edit | Edit Acceleration option in the Bookings Last 24 Hrs report:

3. Enable 1 Day acceleration as seen in the following screenshot and click on Save:

4. To check the progress of your report's acceleration, click on Settings | Report acceleration summaries:

If Eventgen has not been running for more than a day, the report acceleration build may not execute because there is not enough data for the scope of the acceleration. To verify the report acceleration, ensure that Eventgen has been running.

Scheduling options

No matter how advanced and well-scaled your Splunk infrastructure is, if all scheduled reports and alerts are running at the same time, the system will start experiencing performance issues. Typically, you will receive a Splunk message saying that you have reached the limit of concurrent or historical searches. There are only a certain number of searches that can be run on fixed CPU capacity for each Splunk server or collection of servers. A common problem a Splunk administrator will inevitably face is how to limit the number of searches running at the same time. One way to fix this is to throw more servers into you Splunk environment, but that is not a cost-efficient way.

It is important to properly stagger and plan scheduled searches, reports, alerts, dashboards, and so on, ensuring they are not all running at the same time. In addition to the schedule time...

Summary indexing

In a matter of days, Splunk will accumulate data and start to move events through the bucketing process. With the millions or billions of events that are typical with a robust Splunk implementation, you can start to understand how searches run over long-time horizons can slow down.

There are two ways to circumvent this problem. In addition to search acceleration, completed earlier in this chapter, faster search results on large amounts of data can be achieved through summary indexing.

With summary indexing, you run a scheduled search and output the results into a different index, often called summary. The result will only show the computed statistics of the search. This results in a very small subset of data that will be much faster to retrieve and report on than going through a large set of detailed event records and summarizing results on the fly. This concept...

Summary

In this chapter, you learned how to enhance data in three ways: classifying data using Event Types, normalizing data using Tags, and enriching data using Lookups. You also learned some reporting and alerting techniques, along with report acceleration. You were introduced to the powerful Cron expression, which allows you to create granularity on your scheduled searches, as well as stagger execution times. Finally, you were introduced to the concept of summary indexing, allowing searches against pre-aggregated and stored historical data in the interest of performance.

In the next chapter, Chapter 5, Dynamic Dashboarding, you will go on to learn how to do visualizations.