Chapter 3. On-boarding Data in Splunk

This chapter will detail the most important aspect of Splunk, that is, adding data to Splunk. We will go through the newly added feature in Splunk 6.3 of JSON and REST API format of IoT event collections, HTTP Event Collector, and then, we will cover the various interfaces and options to on-board data on Splunk. We will also study how to manage event segmentation and improvise the data input process.

The following topics will be covered in this chapter:

• Deep diving into various input methods and sources

• Adding data to Splunk—new interfaces

• Data processing

• Managing event segmentation

• Improving the data input process

Deep diving into various input methods and sources

Splunk supports numerous ways to ingest data on its server. Any data generated from a human-readable machine from various sources can be uploaded using data input methods such as files, directories, and TCP/UDP scripts which can be indexed on the Splunk Enterprise server and analytics and insights can be derived from them.

Data sources

Uploading data on Splunk is one of the most important parts of analytics and visualizations of data. If data is not properly parsed, timestamped, or broken into events, then it can be difficult to analyze and get proper insight on the data. Splunk can be used to analyze and visualize data ranging from various domains, such as IT security, networking, mobile devices, telecom infrastructure, media and entertainment devices, storage devices, and many more. The machine-generated data from different sources can be of different formats and types, and hence, it is very important to parse data in the best format to...

Adding data to Splunk – new interfaces

Splunk Enterprise introduced new interfaces to accept data that is compatible with constrained resources and lightweight devices for Internet of Things. Splunk Enterprise version 6.3 supports HTTP Event Collector and REST and JSON APIs for data collection on Splunk.

HTTP Event Collector is a very useful interface that can be used to send data without using any forwarder from your existing application to the Splunk Enterprise server. HTTP APIs are available in .NET, Java, Python, and almost all the programming languages. So, forwarding data from your existing application that is based on a specific programming language becomes a cake walk.

Let's take an example, say, you are a developer of an Android application, and you want to know what all features the user uses that are the pain areas or problem-causing screens. You also want to know the usage pattern of your application. So, in the code of your Android application, you can use REST APIs to forward...

Data processing

Data processing plays a very important role in parsing and enriching data to create insights faster and visualize data with the required analytics. Data processing basically includes event, timestamp, and host configuration.

Event configuration

Any data uploaded on Splunk is termed as an event. An event can be anything from a log activity, error logs, usage logs, to machine-generated data from devices, servers, or from any other sources. Events are used to create visualization and get insight about the source in the Splunk environment. So, it is required to process the events properly, depending on the data and source. The processed events' settings and configurations can be stored later in a source type.

Character encoding

Splunk supports many languages to support internationalization of Splunk Enterprise. The default character's set encoding on Splunk Enterprise is UTF-8, whereas it has inbuilt support for various other encoding available internationally. If the data is not...

Managing event segmentation

Splunk breaks the uploaded data into events. Events are the key elements of Splunk search that are further segmented on index time and search time. Basically, segmentation is breaking of events into smaller units classified as major and minor. Segmentation can be explained with the help of the following example.

The complete IP address is a major segment, and a major segment can be further broken down into many minor segments, as shown in the following screenshot:

It is very important to configure event segmentation, as index-time segmentation affects storage size and indexing speed, and search-time segmentation affects the search speed and ability to create searches based on the result of searches on Splunk Web; depending on the need, specific types of segmentation can be configured. Splunk even provides the facility to apply event segmentation on a specific host, source, or source type.

The following are three types of event segmentation that can be configured...

Improving the data input process

Data input is a very important process before you generate insight and visualizations from data. So, it is very important that the data is indexed, parsed, processed, and segmented properly. It may not be the case that the first approach/setting the user applies is the best, and there may be a need for a trial-and-error method to find the best settings for the data of those types for which settings are not available, by default, in Splunk.

It is always advisable to first upload small amount of data on a test index on a development server of Splunk. Once the data is available on Splunk in the correct format of events in which queries can result in the required visualizations, then the input can be forwarded to the correct index and source on the production server.

Many times, it happens that when you are testing and trying to upload the same file more than once to try different settings of event configuration, Splunk may not index the file, as the filename or...

Summary

In this chapter, we walked through various data input methods along with various data sources supported by Splunk. We also looked at HTTP Event Collector, which is a new feature added in Splunk 6.3 for data collection via REST to encourage the usage of Splunk for IoT. We studied data processing, event segmentation, and ways by which we can improve the data input process. In the next chapter, we will cover how to create analytics and provide meaningful insight over the data uploaded on Splunk.