Chapter 8. Being Proactive – Creating Alerts

In this chapter, we will learn about alerting capabilities within Splunk. You will learn about:

• Alerting on abnormal web page response times

• Alerting on errors during checkout in real time

• Alerting on abnormal user behavior

• Alerting on failure and triggering a scripted response

• Alerting when predicted sales exceed inventory

Introduction

Throughout the previous chapters in this book, you created a great deal of Splunk searches, including historic searches that look back over a period of time and real-time searches. In this chapter, you will learn about alerting—arguably, one of Splunk's most powerful features.

A key part of gaining complete operational intelligence is the ability to be proactive rather than reactive. Periodic, ad hoc searching of the data for certain conditions might provide some operational insight, but a better approach would be to continually monitor the data and know immediately when certain conditions are met. For example, instead of reacting to a network outage after it has occurred, it would be better to proactively look for the factors that could lead to a network outage and prevent it from occurring in the first place. It is this type of proactive approach that Splunk's alerting functionality allows for.

In this chapter, we will continue to build our Operational Intelligence application...

Alerting on abnormal web page response times

It is important that our web application remains responsive for users. Sites that lag frequently put off users and can result in them going elsewhere or lost sales. In Chapter 2, Diving into Data – Search and Report, you completed a recipe that analyzes average response times over a given period. In this recipe, you will create a scheduled alert to identify response times that are abnormal (that is, not within a normal range).

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with navigating the Splunk user interface. You should also have configured the e-mail settings on your Splunk to enable the delivery of e-mail alerts.

Note

Should you be running your Splunk server on a desktop PC for the purposes of this book and wondering how to configure the e-mail settings, there is a good Splunk blog posting on configuring...

Alerting on errors during checkout in real time

A very powerful feature of Splunk is the ability to trigger alerts based on specific conditions in real-time events. From the perspective of operational intelligence, real-time alerting provides the ability to be notified of something that requires immediate action. Real-time alerting in Splunk is based upon an underlying real-time search.

In this recipe, you will create a real-time alert that will trigger anytime there is an error during the checkout stage of our online store. The checkout stage in the purchasing process is where the payment details are submitted by the customer and our sales transactions ultimately occur. Errors here can result in lost sales revenue and lost customers. It is, therefore, important to immediately understand if errors occur such that they can be remediated as soon as possible.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1...

Alerting on abnormal user behavior

In this recipe, you will write a relatively simple real-time per-result type of alert to look for abnormal user behavior. The abnormal behavior you will be looking for would be successful payments that did not go through the checkout process.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with navigating the Splunk user interface. You should also have configured the e-mail settings on your Splunk server to enable the delivery of e-mail alerts.

How to do it...

Follow the steps in this recipe to create an alert when abnormal user behavior occurs:

1. Log in to your Splunk server.

2. Select the Operational Intelligence application.

3. In the Search bar, enter the following search over Last 24 hours:

   index=main sourcetype=log4j requestType=checkout (numberOfItems>10  OR total>3000)
   | table ipAddress, numberOfItems, total, invoice...

Alerting on failure and triggering a scripted response

By now, you have used every different type of alert available and many of the more common alert actions such as e-mailing. However, one extremely powerful alert action feature we are yet to touch upon is the ability to execute a script when an alert triggers.

In this recipe, you will create a simple real-time per-result alert that triggers when any 503 HTTP web server errors are detected. Upon triggering, the alert will execute a script that will write the details of the event to a local file on the server.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with navigating the Splunk user interface.

How to do it...

Follow the steps in this recipe to create an alert on failure and a scripted response:

1. The first thing to do is to write the script that Splunk will execute. Splunk is able to output a number...

Alerting when predicted sales exceed inventory

In this final recipe, you will create a scheduled alert type that triggers when predicted sales are expected to exceed the levels of inventory levels on hand. This type of information is a key perspective of operational intelligence, as by knowing ahead of time that we might be running low on inventory, we might have time to order more before we actually run out.

Getting ready

To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with navigating the Splunk user interface and have a good command over the Splunk search language as you have completed the earlier recipes in this book. You should also have configured the e-mail settings in your Splunk server to enable the delivery of e-mail alerts.

Additionally, this chapter relies on an inventory lookup implemented in the Looking up inventory from an external database recipe in Chapter...

Summary

The key takeaways from this chapter are as follows:

• There are three different types of alerts in Splunk: scheduled alerts, per-result alerts, and rolling-window alerts

• Alerts are based-off underlying historical or real-time searches

• Alerts are triggered based on user-specified conditions and can be throttled as required

• Alerts have a number of different actions that can be performed when an alert is triggered, including sending an e-mail and executing a script

• Alerts play a critical part in gaining proactive operational intelligence

• Alerts can be used for relatively simple use cases such as detecting errors or much more complex use cases such as predicting future sales