The app key-value store

The app key-value store is a feature provided by Splunk Enterprise to manage and maintain the state of the application. Using an app key-value store, users can save and retrieve data from Splunk apps.

System requirements

The app key-value store feature is only available in the 64-bit distribution of Splunk Enterprise. It is not available in the 32-bit version of Splunk. It uses the 8191 port by default, but it can be configured from Server.conf located at $SPLUNK_HOME\etc\system\local by modifying the [kvstore] code block.

Uses of the key-value store

The following are some of the uses of a key-value store:

• It can be used to manage the app state of the user interface by storing the session/application state information

• It creates a checkpoint of the uploaded data in case of modular inputs

• It enlists the environment variable used, accessed, or modified by users

• It is the metadata storage of the user

• It caches results from search queries

Components of the key-value store

The key-value store saves data in the collections of the key-value pair. The key-value store files are located on the search heads. The following are the various components of the key-value store:

• Collections: Collections are containers for data storage similar to a database table.

• Records: Records store the entry of data in the collection.

• Fields: Fields contain the value of data in the JSON format file. Fields correspond to the key name similar to columns in the database table.

• _key: This is the reserved field that contains a unique ID for each record. It is an autogenerated field that is not explicitly specified.

• _user: This is also a reserved field that is used to map the user ID of each record.

• Accelerations: This is used to improve search performance that contains the accelerated fields.

Let's take a look at how to create a key-value store collections via a config file. To use a key-value store, we need to create a key-value store collection using the following steps:

1. Create a collections.conf file in the application's default or local directory, as follows $SPLUNK_HOME\etc\apps\APPNAME\default\collections.conf or $SPLUNK_HOME\etc\apps\APPNAME\local\collections.conf.

2. Modify collections.conf by specifying the name of the collection and optionally, the schema of the data. Listed in the following sublist is the description of the parameters which need to be configured in collections.conf file:

   • [collection_name]: This is the collection name

   • enforceTypes: This is set to True or False to enforce the data types of values when inserting them into the collection.

   • field.name: This is an optional field. The available data types are string, time, Boolean, and number. If the data type is not set explicitly, then it is set to JSON.

Any change in collections.conf needs a restart of the Splunk instance to apply the changes on the search heads. Refer to the following example for better understanding:

[AndroidCollections]  #collection_name

The screenshot that follows shows a code snippet of the sample JSON data:

The following screenshot is the code snippet of the enforce data type for the preceding JSON data:

The following screenshot shows the sample code snippet for hierarchical JSON data:

The following screenshot shows how a data type can be enforced on hierarchical data using a dot (.) notation:

Managing key-value store collections via REST

The Splunk REST API can be used to create, read, delete, update, and manage key-value store data and collections. The Splunk REST API accesses Splunk via the management port (by default, 8089). The following are the REST endpoints for the key-value store:

• storage/collections/config:

  • GET: This fetches a list of collections in a specific app

  • POST: This creates a new collection in a specific app

• storage/collections/config/{collection}:

  • GET: This fetches information about a specific collection

  • DELETE: This deletes a collection

  • POST: This updates a collection

• storage/collections/data/{collection}:

  • GET: This fetches records from a specific collection

  • POST: This inserts a new record into a specific collection

  • DELETE: This deletes all records from a specific collection

• storage/collections/data/{collection}/{id}:

  • GET: This fetches records in a collection by a key ID

  • POST: This updates records in a collection by a key ID

  • DELETE: This deletes a record in a collection by a key ID

• storage/collections/data/{collection}/batch_save:

  • POST: This runs one or more save (insert and replace) operations in a specific collection

Examples

There are various notations used in the following examples, such as username, password, IPAddress, and others. Users need to replace them with their own corresponding values to execute the examples. The following are the examples:

• Fetching a list of collections for an android app:

  curl -k -u username:password \ 
  https://IPAddress:8089/servicesNS/nobody/android/storage/collections/config

• Creating a new collection called AndroidCollections in the android app:

  curl -k -u username:password \ -d name= AndroidCollections \ 
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config

• Defining a collection schema:

  curl -k -u username:password \ 
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections \ 
  -d field.Devicename = string \ 
  -d field.DeviceID = number \
  -d field.DeviceInfo.DeviceBuild = string \
  -d field.DeviceInfo.DeviceAndroidVersion = string 

• Adding data of the hierarchical JSON format to a collection:

  curl -k -u username:password \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections \
  -H 'Content-Type: application/json' \
  -d '{ "Devicename" : "Test Device", "DeviceID" : 9661, "DeviceInfo" : { "DeviceBuild" : "Test build 9661C", "DeviceAndroidVersion" : "Marshmallow 6.0", "DeviceIMEI" : 12345678909876, "DeviceMAC" : "AA:BB:CC:DD:EE:FF" }} '

• Getting all data from the collection:

  curl -k -u username:password \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections

• Getting a specific range of records from collections, for example, records from 10 to 15:

  curl -k -u username:password \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections?sort=Devicename&skip=10&limit=5

• Getting a record of a specific key ID:

  curl -k -u username:password \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections/KEYID

  Where the key ID is the unique _key of collections for which the record is to be fetched.

• Deleting the record of the specific key ID:

  curl -k -u username:password –X DELETE \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections/KEYID

• Deleting all records of the AndroidCollections collection:

  curl -k -u username:password –X DELETE \
  https://IPAddress:8089/servicesNS/nobody/android/storage/ collections/config/ AndroidCollections

Replication of the key-value store

In case of a distributed environment, the key-value store can be replicated to a large number of search heads by enabling replication. By default, the key-value store is not replicated to indexers in distributed deployment of Splunk.

To enable replication, the collections.conf file is to be modified and we need to add replicate = true to the file.