Redis is a very popular, memory-based, lightweight key-value database. Strictly speaking, Redis is a data structure server, according to Matt Stancliff (@mattsta
), one of the most important contributors to Redis (https://matt.sh/thinking-in-redis-part-one). The author of Redis, Salvatore Sanfilippo (@Antirez
), first called it Redis, which stands for REmote DIctionary Server because Redis natively implements various kinds of data structures in memory, and provides a variety of APIs for manipulating these data structures. More importantly, Redis supports high-performance command processing, high availability/scalability architectures, and also data persistence features as a long running data store service.
With the development of high-concurrency and low-latency systems, the use of Redis is becoming more and more widespread. Redis has been ranked in the top ten of the DB-Engine complete ranking (https://db-engines.com/en/ranking) since 2017. Before that, it was ranked at the top of the key-value stores ranking (https://db-engines.com/en/ranking/key-value+store) for quite a long time.
The goal of this chapter is to guide readers to quickly set up a simple Redis instance and learn about some common operations, such as starting, connecting, and shutting down a Redis Server. Retrieving basic information from a Redis Server is also introduced. Moreover, it's essential to have a better understanding of the event model and communication protocol of Redis before learning Redis. The last two sections of this chapter include a detailed discussion of the event model and protocol of Redis.
Redis has an active community on GitHub. Large numbers of pull requests have been made and merged during the years, and the author, Antirez, has always given a timely response on the issues section in GitHub. Hence, the release cycles of Redis are very rapid. From the early versions, 2.6/2.8 to 3.0/3.2, which were widely used, and then to the latest 4.x version, each release of Redis offers some essential enhancements and bug fixes. So using the latest version of Redis, if possible, is one of the best practices. In this book, we are adopting the latest version of Redis 4.0.1.
Redis is an open software written in pure C language so that we can install it by compilation. Major operating systems also include Redis binary packages in their software repository, although the Redis version is often a little out of date.
You can find the download link and basic installation steps at https://redis.io/download. If you would like to build Redis by compiling source code in Linux/Unix/macOS, both the gcc compiler and C Standard Library libc are needed in your environment. When it comes to OS repository installation, all you need are an internet connection and the correct repository configuration.
We will demonstrate the compilation installation of Redis in Ubuntu 16.04.2 LTS (Xenial Xerus). The downloading and building steps are as follows:
$ sudo apt-get install build-essential
$ mkdir /redis
$ cd /redis
$ wget http://download.redis.io/releases/redis-4.0.1.tar.gz
$ tar zxvf redis-4.0.1.tar.gz
$ cd redis-4.0.1
$ mkdir /redis/conf
$ cp redis.conf /redis/conf/
$ cd deps
$ make hiredis lua jemalloc linenoise
$ cd ..
Due to the differences among various operation systems and libraries installed on it, the aforementioned steps will be required when errors occur indicating some dependencies are not satisfied. For example, you may encounter the error message:zmalloc.h:50:31: fatal error: jemalloc/jemalloc.h: No such file or directory.
This step is not a must for most environments, if nothing about dependencies goes wrong.
$ make
If everything goes well, the following message will be shown. It means that the compilation has been done successfully:
It's a good idea to run 'make test' ;)
make[1]: Leaving directory '/redis/redis-4.0.1/src'
$ make PREFIX=/redis install
The following messages represent the success of installation:
/redis
directory and verify that the Redis binary files have been generated:$ ls /redis/bin
redis-benchmark redis-check-aof redis-check-rdb redis-cli redis-sentinel redis-server
Congratulations! You have completed the Redis compilation installation.
Compared to the compilation installation, using apt-get
in Ubuntu to install Redis is much easier. Let's take a look:
$ sudo apt-get update
$ sudo apt-get install redis-server
$ which redis-server
When it comes to the Redis version selection, bear in mind that Redis follows the standard practice of versioning, which is major.minor.patch level. An even-numbered minor stands for a stable release, while an odd-numbered minor means it's an unstable version, although there are a few versions using an odd minor for Redis.
The differences between building Redis by compiling and building via a software repository, are that the former can add optimization or debugging options when compiling, and also own the flexibility of specifying the installation location during installation.
After installation, there are some executable files in the bin
directory. Their description and remarks are shown in the following table:
File name | Description | Remarks |
| Redis Server |
|
| Redis Sentinel | A soft link for |
| Redis Console Tool |
|
| Redis RDB Check Tool |
|
| Redis Append Only Files (AOF) Check Tool |
|
| Redis Benchmarking Tool |
|
For Windows, you can obtain the Redis release of Windows, which the Microsoft Open Technologies group used to maintain at: https://github.com/MicrosoftArchive/redis/releases.
Just download the .msi
executable file and give it a double-click to install, keeping the default configurations.
For macOS, there is no big difference from the procedures in Linux. You can also install Redis by issuing the command, brew install redis
on macOS.
Before accessing Redis, the Redis Server must be started in a proper way. Similarly, under certain circumstances, you have to stop the Redis service. This recipe will show you how to start and stop a Redis Server.
You need to finish the installation of the Redis Server, as we described in the Downloading and installing Redis recipe in this chapter.
The steps for starting and shutting down a Redis Server are as follows:
$ bin/redis-server
Your server should now start up as shown in the following screenshot:
$ bin/redis-server conf/redis.conf
init.d
script:$ /etc/init.d/redis-server start
redis-server
as a daemon in the background at start up, you can edit the configuration file and set the daemonize
parameter to yes
and start with this configuration:$ vim conf/redis.conf daemonize yes $ bin/redis-server conf/redis.conf
The message Configuration loaded
shown in the following screenshot indicates the configuration has already taken place:
Kill
+ PID
(if you run Redis in the background) to stop the Redis service:$ kill `pidof redis-server`
redis-cli
:$ cd /redis
$ bin/redis-cli shutdown
init.d
script, in case you installed it from the repository of the operating system:$ /etc/init.d/redis-server stop
The term instance in Redis represents a redis-server
process. Multiple instances of Redis can run on the same host, as long as they use different configurations, such as different binding ports, data persistence paths, log paths, and so on.
Starting and stopping the Redis instance are basic operations. There is not much to note when starting Redis, but for a data service, stopping a Redis service deserves more attention, because as a data store service, it is of great importance for you to learn how to stop the Redis Server gracefully in order to maintain data integrity.
The reason why using the shutdown
command to stop Redis is highly recommended is that if you care about data integrity and have already set persistence for Redis to save your data in memory to disk (the persistence of Redis will be discussed in Chapter 6, Persistence), issuing the shutdown
command not only terminates the process, but also takes a series of other actions.
First, the redis-server
will stop all the clients, and then one persistence action will be performed if the persistence has been enabled. Afterwards, it will clean the .pid
file and socket
file if there are any, and finally quit the process. By adopting this strategy, Redis does its best to prevent any data loss. Conversely, if the kill
command is used rudely, to terminate the redis-server
process, data may get lost because it has not been persisted before the server is shut down.
It should be noted that using kill
or other process management tools to send a SIGTERM
signal (15 signal) to the Redis process is basically equivalent to the shutdown
command for gracefully stopping the redis-server
.
Configuration parameters can be added to the command redis-server
while starting, which is quite useful when deploying multiple instances on a single host. We can have a single configuration file of common configuration parameters used by multiple instances on the same host. Meanwhile, the unique configuration parameters of each instance can be passed in the command line on startup. This way, the cost of maintaining multiple configuration files is eliminated, and instances can be distinguished easily via ps
or other system commands.
In addition, you can manage your Redis instance using process management tools such as systemd, supervisord, or Monit, which can also prevent you from messing up when you deploy multiple instances on a single host. All we need to pay attention to are the startup configuration parameters mentioned previously and exit signal handling mechanisms.
In the development and maintenance of Redis, the redis-cli
in the bin
directory is the most commonly used tool. This section gives a brief description of its usage so that readers can get a brief idea of how to connect to and use Redis with redis-cli
.
You need an up-and-running Redis Server, as we described in the Starting and shutting down Redis recipe in this chapter.
The steps for connecting to Redis using redis-cli
down a Redis Server are as follows:
redis-cli
:$ bin/redis-cli
127.0.0.1:6379>
The pattern of the preceding prompt is IP:port
, indicating redis-cli
has connected to this Redis instance successfully.
foo value1
, bar value2
:127.0.0.1:6379> set foo value1
OK
127.0.0.1:6379> set bar value2
OK
127.0.0.1:6379> get foo
"value1"
127.0.0.1:6379> get bar
"value2"
shutdown
command:$ bin/redis-cli
127.0.0.1:6379> shutdown
not connected>
not connected
. Then, we quit from redis-cli
and make the connection again with redis-cli
. The following error message will be shown:not connected>quit
$ bin/redis-cli
Could not connect to Redis at 127.0.0.1:6379: Connection refused
Could not connect to Redis at 127.0.0.1:6379: Connection refused
not connected>
By default, redis-cli
connects to a Redis instance running on localhost at default port 6379
. You can also specify the hostname/IP address the Redis Server is running on with the -h
option. Just make sure that the network connectivity between the redis-cli
side and the Redis Server side has no problem.
redis-cli
allows you to specify the port with the -p
option, if your Redis Server is not running on the default port 6379
. This option is also useful if you would like to connect to multiple Redis instances with different binding ports on the same host.
Also, if a Redis instance is protected by password, the -a
option can be used to set the password when connecting to Redis.
In addition, if a Unix socket
file is enabled in Redis, you can connect to the Redis Server simply by using the -s
option.
It is often necessary to do some data prototype verification before hooking up your application to Redis. redis-cli
is a very useful tool for this. It provides an interactive command-line interface for you to quickly verify your data design. In the daily maintenance of Redis Server, redis-cli
also offers a set of commands, including obtaining the metrics, manipulating system states, and performing configuration settings.
redis-cli
The most comprehensive and important information about a Redis instance can be obtained with redis-cli
with the INFO
command. In this section, we will see how to use the INFO
command to fetch these essential statistics.
You need an up-and-running Redis Server as we described in the Starting and shutting down Redis recipe in this chapter.
Follow these steps to get server information of Redis:
INFO
command:$ bin/redis-cli
127.0.0.1:6379> INFO
The result looks as follows:
# Server
redis_version:4.0.1
...
# Clients
connected_clients:1
...
# Memory
used_memory:828352
used_memory_human:808.94K
used_memory_rss:9420800
used_memory_rss_human:8.98M
...
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1504223311
...
# Stats
total_connections_received:1
total_commands_processed:1
instantaneous_ops_per_sec:0
...
# Replication
role:master
connected_slaves:0
...
# CPU
used_cpu_sys:0.01
used_cpu_user:0.00
...
# Cluster
cluster_enabled:0
<section>
parameter.For example, you will get the memory metrics by sending INFO memory
in redis-cli
:
redis-cli INFO
directly in a shell command line. In this way, it is quite handy for piping the output to a script for metrics analysis or performance monitoring.The INFO
command gives you all the current Redis metrics and the pattern of each metric is metric-name: metric-value
, which can be easily parsed afterwards.
The following table summarizes the description of each section returned by INFO
:
Section name | Description |
| Basic information about the Redis Server |
| Status and metrics of client connections |
| Overall memory consumption metrics |
| Data persistence related states and metrics |
| General statistics |
| Status and metrics of master-slave replication |
| CPU consumption |
| Status of Redis Cluster |
| Database related statistics |
It is a common practice to build a Redis monitor application by getting information from the INFO
command periodically.
INFO
for Redis maintenance operation and troubleshootingINFO
Redis, known for its high performance, makes the most of a single thread, non-blocking I/O model to process requests rapidly. Therefore, understanding the event model of Redis is essential. As a taster for readers to understand the model, this recipe first shows an echo server
demo program built on the asynchronous event library of Redis (ae
library). Then we provide important insights into the event processing model of Redis by analyzing the core snippet of source code in Redis.
This recipe includes a lot of C programming practices. So if you feel unfamiliar with the C language, you can skip this recipe if you wish and it won't bother you too much as you read along.
This recipe involves source code building and debugging. So you need to finish the Downloading and installing Redis recipe in this chapter first. For better illustration, an IDE that supports the C programming language is needed. The IDE we use here is CLion in Ubuntu Desktop 16.04.3 LTS. While the CLion IDE is not free, a 30-days free trial is enough for us.
You should also prepare the C compiler and development environment. In Ubuntu, you can issue the following command to install the related packages:
$ sudo get update && apt-get install build-essential
After installation, you should make sure the version of CMake is 3.5 or above:
$ cmake --version
cmake version 3.5.1
CMake suite maintained and supported by Kitware (kitware.com/cmake).
To understand the Redis Event Model, take the following steps:
~$ mkdir coding; cd coding
~/coding$ tar xzvf redis-4.0.1.tar.gz
~/coding$ cd redis-4.0.1/deps/
~/coding/redis-4.0.1/deps$ make lua linenoise hiredis
The following screenshot indicates the dependencies have been built successfully:
/redis/coding/
:~/coding$ wget https://download.jetbrains.8686c.com/cpp/CLion-2017.2.2.tar.gz
~/coding$ tar zxvf CLion-2017.2.2.tar.gz
redis-server
building and debugging.redis-4.0.1
directory:~/coding$ tar xzvf echodemo.tar.gz -C redis-4.0.1/
Make sure the following files, CMakeLists.txt
and echodemo
exist in the redis-4.0.1
directory:
~/coding/clion-2017.2.2$ bin/clion.sh
Import Project from Sources
and select the redis-4.0.1
sub-directory in the coding
directory.OK
button and then select Open Project
to open a Redis project.Build All
configuration in the upper-right corner and click the Run
button:Ignore the error of not specifying an executable target, and click Run
.
The following logs show you have managed to build both the echo-server/client
demo program and redis-server
(my-redis-server
in this example):
/redis/coding/clion-2017.2.2/bin/cmake/bin/cmake --build /redis/coding/redis-4.0.1/cmake-build-debug --target all -- -j 6
Scanning dependencies of target ae
Scanning dependencies of target my-redis-server
[ 1%] Building C object CMakeFiles/ae.dir/src/zmalloc.c.o
...
[ 17%] Building C object CMakeFiles/my-redis-server.dir/src/blocked.c.o
Scanning dependencies of target echo-server
...
[ 25%] Built target echo-server
[ 26%] Building C object CMakeFiles/my-redis-server.dir/src/config.c.o
...
[ 31%] Building C object CMakeFiles/my-redis-server.dir/src/defrag.c.o
[ 32%] Linking C executable echo-client
[ 32%] Built target echo-client
[ 98%] Building C object CMakeFiles/my-redis-server.dir/src/zmalloc.c.o
[100%] Linking C executable my-redis-server
[100%] Built target my-redis-server
/redis/coding/redis-4.0.1$ cmake -- The C compiler identification is GNU 5.4.0 ... -- Configuring done -- Generating done -- Build files have been written to: /redis/coding/redis-4.0.1 /redis/coding/redis-4.0.1$ make Scanning dependencies of target my-redis-server [ 1%] Building C object CMakeFiles/my-redis-server.dir/src/adlist.c.o [ 2%] Building C object CMakeFiles/my-redis-server.dir/src/ae.c.o ... [ 85%] Building C object CMakeFiles/my-redis-server.dir/src/zmalloc.c.o [ 86%] Linking C executable my-redis-server [ 86%] Built target my-redis-server Scanning dependencies of target ae ... [ 92%] Built target ae Scanning dependencies of target echo-server ... [ 96%] Built target echo-server Scanning dependencies of target echo-client ... [100%] Built target echo-client
echo-server
, echo-client
, and my-redis-server
in the cmake-build-debug
directory under redis-4.0.1
:echo-server
configuration in the upper-right corner, and click the right arrow button to get it running:echo-server
using nc
(Netcat):~/coding/redis-4.0.1/cmake-build-debug$ nc 127.0.0.1 8000
Hello Client!
will be printed out when the server has started successfully:The server logs the connection:
Hello, please echo!
in nc
and press Enter to send it out. The same message will be echoed back:On the server side, it logs the data, which will be sent back to the client later:
echo-server
. A similar result will be obtained on both nc
and server side, as shown in the following screenshot:redis-server
(called my-redis-server
in this example) to dig into the source code in almost the same way as the echo-server
example. The only thing you have to change is to select the my-redis-server
run/debug configuration profile:As mentioned earlier, Redis takes great advantage of the non-blocking, multiplexing I/O model in its main processing single thread, although there are some circumstances where Redis spawns threads or child processes to perform certain tasks.
Redis contains a simple but powerful asynchronous event library called ae
to wrap different operating system's polling facilities, such as epoll
, kqueue
, select
, and so on.
So what's a polling facility of an operating system? Let's take a real-life scenario to illustrate it. Imagine you have ordered five dishes in a restaurant. You have to fetch your dishes by yourself at a waiting window, and you want to get them as soon as possible once the dishes get done because you are hungry. You may have three strategies for this scenario:
Considering the time and efforts it takes, the third option is the best one, obviously.
The polling facility of an operating system works in a similarly way as the third option does. For the simplicity of this section, we only take the epoll
API in Linux as an example. First, you can call epoll_create
to tell the kernel that you would like to use epoll
. Then, you call epoll_ctl
to tell the kernel the file descriptors (FD) and what type of event you're interested in when an update occurs. After that, epoll_wait
gets called to wait for certain events of the FDs you set in epoll_ctl
. The kernel will send a notification to you when the FDs get updated. The only thing you have to do is to create handlers for certain events.
The whole multiplexing process is shown as follows:
I/O multiplexing model
The ae
library in Redis basically follows the preceding procedure to process the requests. In the echo-server
example, we create an event loop by calling aeCreateEventLoop
firstly. Then a TCP server is built via anetTcpServer
for network binding and listening. We call anetNonBlock
to set the non-block I/O action for this socket FD. After that, we specify the acceptance event handler acceptProc
for the socket FD using the event loop created in aeCreateEventLoop.
Once a TCP connection is established, the server will trigger the action in acceptProc.
In acceptProc,
we use anetTcpAccept
to accept the connection request and register readable events of the socket FD for readProc
. Then readProc
gets called, in which we read the data sent to the server and register the writable event of the socket FD. The event loop then receives the writable event to fire the writeProc
to send back the data to the socket client.
Redis works in pretty much the same way as this echo-server
does. In the main function of server.c
, aeCreateEventLoop
, anetTcpServer
, and anetNonBlock
get called in the initServer
method to initialize the server:
The acceptance handler is also set in the initServer
method, as the following screenshot shows:
Once the server is initialized, the aeMain
method gets called:
In the aeMain
method, asProcessEvents
is called to process the events continuously:
It's clear that no thread or sub-process spawns or interacts in the polling process. Therefore, the key benefit of this model is that it's a light context switch I/O model, so that it's not costly for context switching. A number of limitations need to be considered for this processing model. The most common problem you may encounter is latency. In the polling model, Redis won't process any other commands until the one being processed finishes. So keep in mind from now on that an unexpected latency will be the first headache for you when using Redis.
Other polling methods, such as poll, select, and so on, are not discussed here due to the limited area. If you are working on platforms other than Linux, you can debug the source code to learn more about the polling facility on that platform.
As we described in the previous recipe, Redis is merely a non-blocking, I/O multiplexing TCP server that accepts and processes requests from clients. In other words, in spite of the complexity within the Redis Server, you can talk to Redis over the TCP connection in various languages. The term protocol stands for the language used between a server and a client in the networking communication. As for Redis, REdis Serialization Protocol (RESP) is the protocol. In this recipe, we'll see how the RESP works in Redis.
One thing should be noted: Even at first glance, it's a little bit advanced for a beginner to go through this recipe. However, we do believe that learning the RESP, as basic knowledge, is not that difficult and it will be of benefit to you, in that understanding the various kinds of clients and proxies implemented in different programming languages will no longer be a mystery to you.
Of course, you can skip this recipe if you like and it won't bother you too much when reading the following chapters.
You need an up-and-running Redis Server, as we described in the Starting and shutting down Redis recipe in this chapter.
The tool, netcat (nc
) should be installed. In this recipe, we have the nc
command in Ubuntu 16.04.3 LTS for netcat. You can use Cygwin to install netcat if you work under Windows.
To understand the Redis protocol, take the following steps:
PING
command to the Redis Server with netcat:Instead of sending the command PING
in redis-cli
, let's build the command using RESP:
$ echo -e "*1\r\n\$4\r\nPING\r\n" | nc 127.0.0.1 6379 +PONG
SET
and INCR
commands to set an integer and increase it by one:$ echo -e "*3\r\n\$3\r\nset\r\n\$5\r\nmykey\r\n\$1\r\n1\r\n" | nc 127.0.0.1 6379 +OK $ echo -e "*2\r\n\$4\r\nINCR\r\n\$5\r\nmykey\r\n" | nc 127.0.0.1 6379 :2
You may encounter the following error when you send a non-existent command:
$ echo -e "*2\r\n\$3\r\ngot\r\n\$3\r\nfoo\r\n" | nc 127.0.0.1 6379 -ERR unknown command 'got'
Multi-commands can be combined and sent to the Redis Server in a single network transmission:
$ echo -e "*3\r\n\$3\r\nset\r\n\$3\r\nfoo\r\n\$3\r\nbar\r\n*2\r\n\$3\r\nget\r\n\$3\r\nfoo\r\n" | nc 127.0.0.1 6379 +OK $3 bar
There is a big chance that you will go through these commands in tremendous confusion. As we stated in the Getting ready section in this recipe, these commands are the language in which the Redis Server and client talk to each other. It's easy for you to learn them in that there are only five types in RESP.
Let's take a look at each command.
First, we sent *1\r\n\$4\r\nPING\r\n
to the Redis Server. The command begins with an asterisk indicating this is an arrays type.
Look at the following:
1
stands for the size of this array.\r\n
(CRLF) is the terminator of each part in RESP.$4
is the escape character for the $
sign. $4
tells you that the following is a bulk string type of four characters long.PING
is the string itself.+PONG
is the string the PING
command returned. The plus sign tells you that it's a simple string type.The next type we talk about is the integer type. Look at :2
, which is the result returned by the INCR
command. The colon before the number indicates this is an integer.
Sometimes, the server may return an error type message beginning with a minus when a non-existent command has been processed, such as the got
command shown previously.
In addition, for the consideration of performance, you may send multiple commands in a single call to the Redis Server using RESP.
To sum up, the client sends commands to a Redis Server as a RESP array of bulk strings. Then the server replies with one of the five aforementioned RESP types accordingly.
Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.
If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.
Please Note: Packt eBooks are non-returnable and non-refundable.
Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:
If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:
Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.
You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.
Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.
When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.
For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.