Learning RabbitMQ

A typical enterprise will have a number of systems that must typically communicate with each other in order to implement a well-defined business process. A question that is frequently tackled for this reason is how to implement the communication channel between these types of systems? For example, consider the following diagram:

The question mark in the preceding picture denotes the communication media for the six systems that are illustrated. In the diagram, we can think of these separate systems as the components of one large system and the problem stays the same. Before discussing the various alternatives for integration, a number of key factors are considered, as follows:

Let's assume that the systems communicate directly via some kind of remote procedure calls as shown in the following diagram:

This implies that separate communication links must be established between each pair of systems, which leads to tight coupling, a lot of effort to maintain all of the links, reduced scalability, and reduced extensibility (for every new system that is added, a few more communication links with other systems must be created). However, real-time communication requirements might be met with some additional effort to design the communication links.

A second approach is to use a shared file system in order to exchange files between the systems that are being integrated, as illustrated in the following diagram:

A shared file system is used to provide the communication medium. Each system may export data to a file that can be imported and used by other systems. The fact that each system may support its own data format leads to the fact that each system must have a particular mechanism to import data from every other system that it needs to communicate with. This, on other hand, leads to the same problems that are described in the case of direct communication. Moreover, real-time communication requirements might be more difficult to establish and reading or writing data from disk is also an expensive operation.

A third option is to use a shared database as shown in the following diagram:

Here, all the systems should depend on the same database schema. Although this reduces coupling between systems and improves extensibility (new systems must conform to a single database schema in order to integrate with other systems), real-time workload processing is still an issue. Scalability and maintainability depend directly on the type of database that is being used and they could turn out to be weak factors especially if it is a relational database (this may not be the case if NoSQL solutions are applicable for the particular use case).

Messaging comes to the rescue when considering the problems that arise when applying the previous approaches. Consider the following diagram for the Enterprise Messaging System:

A message is the central unit of communication used in enterprise messaging systems. A message typically consists of the following:

The messaging system itself provides mechanisms to validate, store, route, and transform messages that are being sent between the different systems. Each system is responsible for crafting its own message that is transferred via the messaging system (also called the messaging broker) to other systems that are connected to the broker and configured to receive that particular type of message. Each system may create a message in a proper format that is specified by the protocol of the message broker—meaning that the system is only coupled with that particular protocol. If the broker implements a protocol that is based on a well-recognized standard, then this would further decouple the systems from that particular message broker implementation.

Real-time workload processing is typically quite fast as the particular protocol that is implemented by different messaging brokers is optimized to process message data in a reliable and secure manner with minimal overhead. Most messaging solutions provide a number of facilities that allow easy configuration, management, and monitoring; thus, simplifying maintainability. Clustering support is also considered by most implementations due to scalability and reliability requirements and increasing workload demands. Integrating new systems is a matter of implementing a mechanism for direct communication with the message broker.

In case the different systems provide different implementations of messaging protocols (such as REST, SOAP, JSON-RPC, JMX, AMQP, and many others), a messaging system could further provide various adapters for the different protocols as well as extended mechanisms for routing and transformation of different types of messages—this extended functionality also categorizes the message brokers as Enterprise Service Bus (ESB) solutions. One major drawback of an ESB is that it must implement all the communication requirements of all systems that are being integrated by the ESB, otherwise workarounds must be used in order to implement direct communication between the integrated systems (thus, neglecting the usage of an ESB).

The RabbitMQ messaging server is an open source implementation of the AMQP 0-9-1 protocol (Advanced Message Queuing Protocol) that defines how messages should be queued, routed, and delivered in a reliable and secure manner. AMQP 1.0, which is an OASIS (Organization for the Advancement of Structured Information Standards), is not directly supported in the message broker; however, RabbitMQ provides a plugin for AMQP 1.0 (as it is not backward-compatible with AMQP 0-9-1). OASIS is a non-profit organization that works for the development of a number of technology standards. As an open standard, AMQP promotes interoperability among the messaging brokers that implement the protocol. It also defines the delivery semantics for a message, which dictates how many times that message can be sent from one endpoint to another—zero or once, exactly once or multiple times. As a wire protocol, AMQP provides better performance in regard to other messaging protocols such as XMPP (Extensible Messaging and Presence Protocol).

Before we discuss more about RabbitMQ as a message broker, we will introduce some terminologies from the RabbitMQ world that we will use frequently throughout the book:

The AMQP protocol allows a client to establish a one-way logical link to send messages for exchange. Each logical link is a separate AMQP channel that may provide additional options for the reliable transfer of messages. In this regard, a single-client TCP connection to the RabbitMQ server allows multiple AMQP channels of communication. Since AMQP does not provide the capability to retrieve the list of queues, exchanges, or bindings that are defined in the RabbitMQ message broker, client applications must specify the exchange name, queue names, and, optionally, routing information by means of routing keys for particular bindings. AMQP is a programmatic protocol that allows its clients to create and delete exchanges, queues, and bindings when necessary. RabbitMQ addresses some limitations of AMQP by providing custom extensions apart from the fact that the AMQP protocol itself is extensible. In order to simply application development, RabbitMQ provides several exchange types out of the box, as follows:

Receivers can either subscribe to a queue in order to receive messages (also called push-style communication) or request messages on demand from a queue (also called pull-style communication).

Installation

You can download a RabbitMQ distribution package for the operating system of your choice from http://www.rabbitmq.com/Windows.

For Windows operating systems, you have the ability to use the provided RabbitMQ installer (the simpler alternative) or manually install the broker using the provided zip distribution archive (requires additional setup of Windows system paths). We will provide an overview of the installation process for Windows 7 using the installer for Rabbit 3.3.5 (rabbitmq-server-3.3.5.exe) that is quite straight-forward. Initially, the installer checks whether Erlang is installed on the target Windows system and, if it cannot find it, a dialog prompts you to install it, as shown in the following image:

If you click Yes, the dialog redirects you to the official Erlang website. There, you will find the appropriate binaries for your 32-bit/64-bit Windows operating system. Download and install the Erlang 17.3 distribution (compatible with RabbitMQ 3.3.5) for 64-bit Windows (otp_win64_17.3.exe):

File associations can be established and the Erlang documentation can be installed as a part of the Erlang installation process:

The next step is to specify the location to install Erlang:

Finally, you have the option to place an Erlang shortcut in the Start menu folder. After the installation is finished, you can run the RabbitMQ 3.3.5 server installer again:

You can specify in addition that Start menu items and a Windows service should be added along with RabbitMQ server installation. Adding a Windows service for RabbitMQ server is usually recommended as it provides a convenient mechanism to manage a RabbitMQ server instance:

The final step is to specify the location to install the RabbitMQ server. Once the installation is complete, the installer tries to start the RabbitMQ server and, if your Windows firewall is turned on, you might be prompted to allow access to the RabbitMQ server in order to open a port on the target machine (the default port is 5672 for a RabbitMQ server node instance). In order to check whether the RabbitMQ service is running, you can open services.msc from the Windows Run menu and check whether the RabbitMQ service has started. Additionally, you can check whether the RabbitMQ instance node is initiated, by default on port 5672, by executing from the command prompt:

netstat –a

RabbitMQ installation also provides a number of command-line utilities that you can use in order to manage the RabbitMQ instance, which is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory. You can use the rabbitmqctl utility to check the status of a broker, start, or stop it:

rabbitmqctl.bat status
rabbitmqctl.bat stop
rabbitmqctl.bat start

In addition to this, rabbitmqctl provides a number of other commands that can be used to manage the RabbitMQ broker. There is a RabbitMQ management plugin that provides the ability to manage a RabbitMQ broker from a web-based interface and in particular to do the following:

• Manage broker objects such as message queues, users, and permissions

• Send messages to the broker

• Receive messages from the broker

• Monitor and manage connections to the broker

• Monitor broker workload

• Monitor resource usage such as memory, processes, and file descriptors that are used by the broker

These are included in the set of plugins that are installed by default; however, it must be enabled by executing the rabbitmq-plugins utility that is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory from the following command prompt:

rabbitmq-plugins.bat enable rabbitmq_management

After the management plugin is enabled, you have to restart the RabbitMQ server:

rabbitmq-server.bat restart

The management plugin starts an http server, on port 15672, by default in order to verify that the plugin is trying to open http://localhost:15672/ from a browser. You will be prompted to provide a Username and Password in order to login:

By default, RabbitMQ installs with a user with a guest name and guest password that are only available from local host connections to the broker. In the next chapter, we will see how to manage users for a RabbitMQ server.

Linux

For various Linux distributions, there are out-of-the-box packages provided for the RabbitMQ server. Some Linux distributions also provide you with the ability to install the broker directly from a package repository. We will provide an overview of the installation process for Ubuntu 12.04 Desktop edition based on a package repository that we can also download and install directly, a RabbitMQ Debian package for the purpose. To install the broker and enable the management plugin, open a terminal and execute the following command:

echo "deb http://www.rabbitmq.com/debian/ testing main"  | sudo tee  /etc/apt/sources.list.d/rabbitmq.list > /dev/null
sudo wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update
sudo apt-get install rabbitmq-server –y
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart

Tip

Downloading the example code

You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The server installation also installs utilities for the management of the RabbitMQ server—used in the same way as in the Windows installation of RabbitMQ.

The Corporate Social Network (CSN) is a social networking service that is being deployed in an enterprise and allows its users to upload content and interact with each other. In particular, the system allows the user to post blogs, upload files, subscribe to other user profiles (in order to track the activity of other users), and chat with other users. The social network uses RabbitMQ in order to process events that have been triggered by user activity, trigger long-running jobs (such as batch file uploading), and serve as a backbone for the delivery of chat messages from the chat feature of the social network. The following diagram provides a high-level overview of the components of the system:

We will design the system from the very beginning and then start expanding it. In the meantime, we will demonstrate the various capabilities of RabbitMQ by applying them to the extensions of the social network.

In this chapter, we covered the fundamentals of enterprise messaging solutions and discussed the features of RabbitMQ along with the installation process. A brief comparison with other messaging brokers was provided in order to reveal what the strengths and weaknesses of RabbitMQ are compared to the other alternatives. We also introduced a case study project CSN that makes use of RabbitMQ as a messaging solution for propagation of events throughout the system and lays the basis for further demonstrations on the various features of RabbitMQ.

Attempt the following questions: