Instant Redis Optimization How-to: Learn how to tune and optimize Redis for high performance

Redis was started by Salvatore Sanfilippo to improve and extend LLOOGG, which is a real-time website-analytics system. Once Redis became stable enough for production environments and grew in popularity, it turned into a standalone open source project under BSD license, completely sponsored by VMware.

If you are a social or a technology person, you have undoubtedly experienced the fast performance of Redis. For example, Quora uses Redis to provide the front-page feeds and Twitter implements Redis for its deep-structured awareness. The list goes on with high-profile online applications such as GitHub, Stack OverFlow, YouPorn, and Craigslist,.

It is important to know what we can do with Redis and it is equally important to know how it compares with other similar offerings in the market. So, in this section, we will learn how it varies in general and compare it with a few important NoSQL data stores.

The make install command copies all the executables to the/usr/local/bin folder. The configuration files and init scripts are not yet in place to run Redis.

Now we have a working Redis server with the basic configuration.

If you are using Mac OS X, you can install Redis from the source or by using Homebrew. The installation process from the source is the same as we did for Debian. So we will see how to install Redis in Mac using Homebrew. You should have OS X command-line tools installed in the system for Homebrew to work. If you do not have the command-line tools installed, you can download them from the Apple Developer website (http://connect.apple.com/).

Homebrew installs and configures the server. You can test the installation by running redis-cli and issuing commands to the server.

If you are a loyal user of the Windows operating system, you have hit a hurdle. Unfortunately, Redis is not officially supported in Windows 32- or 64-bit systems. But there are various ports available that are not of production quality and can be used only for development purposes. As Redis is not natively supported in Windows, the performance of ported builds is not even close to its performance in Linux. But if you do not have access to Linux systems, the compiled packages can be downloaded from the GitHub page of the officially recommended Windows port (https://github.com/dmajkic/redis/downloads).

As it is not officially supported for Windows, the book will focus on the Linux installation, and using a Linux system is strongly recommended.

Security comes into picture only when Redis is connected to the systems through the network. When you want the clients to connect to Redis from external systems, you need to take extra measures to make sure your data is safe and cannot be exploited. If there is no authentication to the Redis instance, it is very easy to clear the entire data using a single command. In order to prevent this, Redis provides an authentication mechanism using a password.

Uncomment the line that starts with requirepass and replace the word next to it with a strong password. As mentioned in the configuration file, it is very easy to break a simple password using brute force due to the sheer speed of Redis.

requirepass reD1$$er\/erR0ck$

Another security feature provided by Redis is command renaming. For an authenticated client, it is easy to change the configuration of the server using the CONFIG command. To prevent any misuse of commands and to restrict clients from using certain commands, you can use the command-renaming option.

For instance, if you see the command in the configuration file, then the CONFIG command can be renamed to some random sequence—preventing any clients, except internal tools, from using the command.

rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52

You can also kill any command completely by just renaming the command to an empty string.

One of the limits you might want to set is maxclients, which controls how many clients can connect to the server simultaneously. After reaching the maximum connections, the new connections will be ignored. So, set this with caution after considering your use case. This parameter, along with timeout, can be used to limit and control the client's connection to the server.

maxclients 100

Redis encodes the data types into more efficient data structures to optimize the memory space when the size of data in the memory is less than the configured limit. Data types like hashes, lists, sets made of integers, and sorted sets with size less than the configured element size are encoded in an efficient way that uses up to 10 times lesser memory. This is completely transparent to the clients that are performing the operations and can be controlled using the configuration parameters. Redis performs the operation of converting these special encoded types to normal data structures as soon as the data size exceeds the configured value, in a completely transparent manner.

Let us leave the configuration to default. We will discuss how these work and what the significance of these numbers is, in the Optimizing memory recipe.

The advantages of configuring snapshotting in Redis are as follows:

Snapshotting, in spite of the advantages mentioned, has a few limitations that you should be aware of:

Now that we have discussed both the modes of persistence Redis provides us with, the big question is what should we use? The answer to this question is entirely based on our application and requirements. In cases where we expect good durability, both snapshotting and AOF can be turned on and be made to work in unison, providing us with redundant persistence. Redis always restores the data from AOF wherever applicable, as it is supposed to have better durability with little loss of data. Both RDB and AOF files can be copied and stored for future use or for recovering another instance of Redis.

In a few cases, where performance is very critical, memory usage is limited, and persistence is also paramount, persistence can be turned off completely. In these cases, replication can be used to get durability. Replication is a process in which two Redis instances, one master and one slave, are in sync with the same data. Clients are served by the master, and the master server syncs the data with a slave.

Replication setup for persistence

Consider a setup as shown in the preceding image; that is:

In this case, the master does not need to perform any background disk operations and is fully dedicated to serve client requests, except for a trivial slave connection. The slave server configured with AOF performs the disk operations. As mentioned before, this file can be used to restore the master in case of a disaster.

Persistence in Redis is a matter of configuration, balancing the trade-off between performance, disk I/O, and data durability. If you are looking for more information on persistence in Redis, you will find the article by Salvatore Sanfilippo at http://oldblog.antirez.com/post/redis-persistence-demystified.html interesting.

Redis is a single-threaded application. A single thread serves all the requests from the clients. In cases where multiple simultaneous requests are received, the requests are queued and processed sequentially. This might look like a bad idea, as requests may take longer to be processed than expected, but Redis is not perceived as slow due to the very little time taken to complete a single request, and the thread does not get blocked in any I/O operations as all the I/O operations are performed by the forked child processes.

Due to the single-threaded architecture, even when provided with multiple cores, Redis cannot leverage them. So Redis likes processors with larger caches and is neutral towards multiple cores. There is very little chance of the CPU becoming the bottleneck, as Redis is usually memory- or network-bound.

But to make use of multiple cores, we can start multiple Redis instances in the same server using different ports and treating them as different servers. Due to the low memory footprint of Redis (approximately 1 MB per instance), we can run multiple instances without any serious load to the memory.

Apart from the server setup and persistence, even the application's design can affect the performance of Redis. For example:

The next type of data imports millions of records in a short span. In this recipe, we will take a look at how to feed Redis with a huge amount of data as fast as possible. Usually, bulk importing of data into Redis is performed in the following scenarios:

In these cases, the data to be imported is usually huge and has millions of writes. To achieve the import, using a normal Redis client is not a good idea as sending commands in a sequential manner will be slow and we need to pay for the roundtrip. We can use pipelining, but pipelining makes it impossible for the data inserted to be read at the same time, as Redis will not commit the data till all the commands in the pipeline are executed.

*<number of arguments> CR LF
$<number of bytes of argument 1> CR LF
<argument data> CR LF
...
$<number of bytes of argument N> CR LF
<argument data> CR LF

SET samplekey testvalue

*3  - Number of arguments = SET + key + value = 3
$3  - Number of Bytes = 3 - SET has 3 bytes
SET
$8  - Number of bytes in samplekey
samplekey
$9
testvalue - Number of bytes in testvalue

*3\r\n$3\r\nSET\r\n$8\r\nsamplekey\r\n$9\r\ntestvalue\r\n

cat redis-data.txt | redis-cli --pipe

All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1000000

Strings are the most basic data type in Redis, implemented using a dynamic string library written in C. This basic type is binary safe and used to create more complex data types such as lists and sets. A string value can hold up to 512 MB of data. The string value can be anything from the HTML contents of a page to a binary stream of an image file.

There are many useful commands available to use on strings. A few notable ones are INCR, DECR, INCRBY, SET, GET, and APPEND.

Without the complex data types like lists and sets, Redis will be nothing but a MemCached server with persistence. In Redis, lists are nothing but a list of strings arranged based on the order of insertion. This ensures that the data can be accessed in the same order in which it was inserted. Internally, lists are implemented using Linked List, which has both pros and cons. The advantage is that adding an element in the head or tail of the list takes the same time. For example, adding an item to a list with 10 items or 10,000 items takes the same time, but the main disadvantage is that accessing an element in the middle of the list takes longer than accessing elements near the extremes. Being a database, it is critical to be able to add elements to a large list in an efficient way.

The LPUSH command is used to push new elements into the list and the LLEN command outputs the number of elements in the list.

Various operations available for list manipulation make it easy to model queues or stacks. The maximum number of elements that can be stored in a list is 2³²-1.

Redis sets are an unordered data collection of binary safe strings with no duplicate members. You should consider using sets when you have a collection of items and you need to add, delete, or verify the existence of members in a very efficient and fast manner. Another desired behavior is no duplication, and also support for peek and pop of elements (using the SRANDMEMBER and SPOP commands). Sets can also store up to 2³²-1 elements.

Sets also support complex operations like intersections and unions of members on the server side, which lets us perform transformations on the data before we get the output data. SADD is used to add an item into a set.

Sorted sets are introduced as a solution to a few shortcomings in sets, and are the most advanced data type in Redis. Similar to sets, sorted sets are also collections of non-repeating binary safe strings, but with a score associated to the member. The associated scores are used to sort the set from the smallest to the greatest score. We can add, delete, or update elements in a sorted set quickly, because elements are inserted in an order rather than ordered afterwards.

Due to the highly optimized and efficient working of sorted sets, they can be used to maintain leader boards, timestamp range data, or to implement auto-completion with Redis. They make good priority queues and can be used to implement weighted random selections.

Hashes are perfect to represent objects as a map between string fields and string values. Hashes are equivalent to a hash table or a hash map in Java. If we need to store some data related to an object and do not want to perform encoding (in JSON or XML, for example), hashes are the best suited. Hashes can be used to represent linked data structures using references.

We already discussed the special encoding used for small aggregate data types such as hashes, lists, sets, and sorted sets. The purpose of these encodings is to reduce the memory usage at the cost of CPU. These encodings are memory-efficient and take up to 10 times less memory. As soon as the size goes higher than the configured maximum size, the data set will be converted to normal encoding, which is completely transparent to the user.

Redis compiled with a 32-bit target leaves a lesser memory footprint compared to the 64-bit build, as the size of the pointer is small, but such an instance will be limited to 4 GB of maximum memory usage. The persistence files are compatible with both 32- and 64-bit instances and interchangeable without any trouble.

Small hashes take less space in memory due to the special encoding discussed in the last section. So it is good to represent data in hashes whenever possible. For instance, if you want to store some data (ID, name, and price) for a product, use a single hash with all required fields instead of using different keys for each field. In this case, storing in a hash field will take three times less memory than storing the data as different keys.

For example, let us consider we want to cache small HTML fragments for our product catalog, in which the product ID ranges from 1,000 to 10,000. To store the fragments, we can use simple keys, with product ID as the key and the fragment as the value.

SET "Product:1000" "HTML snippet for 1000"
SET "Product:1001" "HTML snippet for 1001"
SET "Product:9875" "HTML snippet for 9875"

By using this method, we will end up creating 9,000 records in Redis. But we can save the same records as hash and save significant memory. Let us see how we can do that.

Every time we perform a SET operation, we will split the key into two parts. The first part will be used as the key and the second part as the hash field. In our case, the key name, Product:1000, will be split as key name Product:10 and field name 00. We will use the following command to set it:

HSET Product:10 00 "HTML snippet for 1000"

After adding all 9,000 items, the hash Product:10 will have a maximum of 100 fields and a maximum of 90 hashes in total, such as Product:11 and Product:12. By doing this, we have only 90 hash keys, each having 100 fields. This is a form of implicit presharding of keys. You can even save memory by keeping the key names short. For example, using pdt:10 is more efficient than using product:10.

If you are planning to store more than 100 fields in a hash, to take advantage of memory optimization, make sure the number is less than the value configured in 6379.conf.

hash-max-zipmap-entries 256
hash-max-zipmap-value 1024

The memory saving achieved using this method is phenomenal, making Redis the most memory-efficient data store. When benchmarking with a proper HTML snippet and 9,000 products, the result of memory usage is as follows:

Auto expiry is used to delete a key after a certain time. We can set a timeout on a key using the EXPIRE command. The error in the expiry is 0 to 1 millisecond in the latest Redis build (Version 2.6). The command is:

EXPIRE key Seconds

After the timeout, the key gets deleted automatically. Such keys are called volatile keys in Redis. The timeout can be cleared using the SET or GETSET commands. A volatile key can be made persistent using the PERSIST command. All other commands such as INCR or HSET, which do not replace the value, will leave the timeout intact. If a key gets renamed using the RENAME command, the timeout is also transferred to the new key name.

The expiry information of the keys is stored in UNIX time and it is heavily dependent on the computer clock. This makes sure the time flows even when Redis is not running.

Internally, Redis performs expirations in two ways, passive and active. In the passive method, the expired key gets deleted when accessed by the client. This is not perfect as the expired keys that are never accessed will keep using memory. In the active mode, Redis performs sampling of 100 records per second and deletes all the expired keys. If the sampling results have more than 25 percent expired keys, the process continues.

Let us discuss how Redis can serve as a perfect caching server, meeting all the basic requirements mentioned previously, in the case of caching an HTML page of the product catalog. The catalog has two types of pages, list pages and product-information pages.

As we have already seen, Redis is ultrafast, having the ability to serve more than 10,000 requests per second with no trouble, when deployed in entry-level servers.

In a typical system, a process checks Redis for a valid cache. If available, it delivers the cached HTML data to the client. If not available in Redis, the process generates the HTML and adds it to Redis as a cache to serve future requests efficiently. The cache needs to be refreshed periodically with fresh data. In this case, the auto-expiry feature in Redis comes in handy. Let us assume that our pricing data is valid for a maximum of three hours. Then, we can set the expiry for the product pages and list pages to three hours (10,800 seconds). This ensures that our cache gets refreshed every three hours.

The most common challenge to any caching system is cache invalidation and invalidating all dependencies. Taking our previous example, we have got a new price for only one product and now both the list page and the particular product page become invalid. It is critical to invalidate the cache before the expiry happens. It is unwise to invalidate the whole cache and recreate them. We need to invalidate only the cache of pages dependent on the changed product. For this purpose, Redis sets can be used to keep track of dependencies and only the dependent cache data need be expired.

Redis stores the cache in the memory. If we have millions of products in our catalog, we do not want to buy and install RAM to support our whole catalog. We want to limit the memory used while removing the least accessed cache to make space for new caches. Redis provides a simple solution to this problem. We can control the maximum memory Redis can use for its data through /etc/redis/6379.conf.

maxmemory <<bytes>>

Other parameters can be used to specify the algorithm to be used to reclaim the memory. The following are as per the Redis documentation:

For our example, volatile-lru makes sense as it will delete the keys with an expiry set and are the least recently used. This way we can limit the amount of memory used by Redis when used as a caching server.

Let us take the example where we are trying to implement a notification system in which we show a notification on receiving a new e-mail, on receiving a chat message, and if someone adds you as a friend. We have three processes working on these three functionalities. Our notification system needs to subscribe to all the three channels and show notifications.

To subscribe to a particular channel, the SUBSCRIBE command should be used if we know the channel name. In this case, our notification client will issue a subscription to all three channels.

SUBSCRIBE email chat addfriend

Once subscribed to any channel, the client cannot issue any other commands but has to wait and read the stream of messages coming back from the server. There are three types of messages possible, which are identified by the first element of the message:

For our command, the response will look something like this:

SUBSCRIBE email chat addfriend
*3
$9
subscribe
$5
first
:1
*3
$9
subscribe
$4
chat
:2
*3
$9
subscribe
$9
addfriend
:3

The format is the same as the Redis protocol, where the first line defines the number of arguments, followed by the number of bytes in arguments, and then the data itself. This confirms our subscription to all three channels. If we publish a message to any of the channels from another redis-cli, we use the PUBLISH command in the following manner:

PUBLISH chat Hi

Our notification client will receive the message stream as follows:

*3
$7
message
$4
chat
$2
Hi

Let us consider a case in which we want to give a provision to the user to stop notifications for chats when busy. In this case, it is possible to unsubscribe from a specific channel by using the UNSUBSCRIBE command.

UNSUBSCRIBE chat
*3
$11
unsubscribe
$4
chat
:2

The reply from the server will confirm the unsubscription, and also mentions that we have two remaining active subscriptions. Due to the simplicity of the Publish/Subscribe functionality, it is very easy to implement.

Redis does not support rollbacks, which is an odd behavior when compared to other relational databases. So, even when an error occurs, Redis continues to execute the other commands. The rollbacking ability was sacrificed to keep the internal design simple and to keep Redis running fast. Another reason for not supporting rollbacks is that errors in Redis commands are only possible because of programming errors.

But there is a caveat. Due to the sequential manner of Redis's command execution in a transaction, it is not possible to perform a read operation during the transaction. Let us consider a case in which we want to write data into Redis based on the value in another key.

    val = GET key1
    MULTI
    if val > 20:
        SET key2 (val+1)
    EXEC

In this case, let us assume the value in key1 was changed (value more than 20) after we read it and before the EXEC was executed. We will end up setting a wrong value to key2. Redis supports Check and Set (CAS) transactions to mitigate this issue. To use CAS, we need to use the WATCH command. The WATCH command provides a form of locking called Optimistic Locking, which is powerful.

    WATCH key1
    val = GET key1
    MULTI
    if val > 20:
        SET key2 (val+1)
    EXEC

In this case, if the value of key1 changes after the WATCH statement and before the EXEC command is executed, the transaction will fail. The client needs to retry again. The WATCH command can be called multiple times to watch multiple keys, or WATCH is called with multiple keys. The WATCH list is cleared once EXEC or UNWATCH is called.

The Redis Slow log is a system to log queries. The slowlog command is used to view the commands that exceeded the configured time in our configuration file. This command points out the queries that are running slowly, which can help us design the system better. Using this command, we can read the slow queries and also reset the queue. The Slow log is stored strictly in the memory, and provides quite rapid logging with little to no performance hit at the cost of extra memory to store the log.

To get the current length of the Slow log, use slowlog len. To read the Slow log, the slowlog get command should be used. This command returns the complete contents of the log. If you want only the latest N number of entries, pass the number to the command as follows:

slowlog get 2

The output format for the log will be:

1)1) (integer) 7
     2) (integer) 1356248799
     3) 15
     4)  1) "SUNIONSTORE"
          2) "Set1"
          3) "set2"
          4) "set3"
2) 1) (integer) 6
     2) (integer) 1356248815
     3) 12
     4)  1) "SINTER"
          2) "Set1"
          3) "set2"
          4) "set3"

The preceding log output has two entries. Every entry consists of four fields, as follows:

To reset the Slow log, use the slowlog reset command. Once the command is executed, the log information is lost forever.

In the latest version (v2.6.13) of Redis, a new debugging tool for developers has been introduced, watchdog. It is designed to track latency problems that could escape analysis using normal tools. This feature is still experimental and should be used in production with caution as a last resort when all other means of debugging fail. It is not advisable to keep the watchdog running for an extended period of time. The output is logged into a logfile that contains low-level reports about the server component. The log can be sent to the Redis community to figure out what caused the block in the server and fix the problem, if any, in the server.

To enable the watchdog, you need to use the CONFIG SET command as follows:

CONFIG SET watchdog-period 700

The preceding command will log any latency issues if the delay is more than 700 milliseconds. The minimum time that can be mentioned is 200 milliseconds.

The following table has the recommended libraries for popular programming languages (which are stable enough to be used in production):