Learning RabbitMQ

4.3 (3 reviews total)
By Martin Toshev
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Introducing RabbitMQ

About this book

RabbitMQ is Open Source Message Queuing software based on the Advanced Message Queue Protocol Standard written in the Erlang Language. RabbitMQ is an ideal candidate for large-scale projects ranging from e-commerce and finance to Big Data and social networking because of its ease of use and high performance. Managing RabbitMQ in such a dynamic environment can be a challenging task that requires a good understanding not only of how to work properly with the message broker but also of its best practices and pitfalls.

Learning RabbitMQ starts with a concise description of messaging solutions and patterns, then moves on to concrete practical scenarios for publishing and subscribing to the broker along with basic administration. This knowledge is further expanded by exploring how to establish clustering and high availability at the level of the message broker and how to integrate RabbitMQ with a number of technologies such as Spring, and enterprise service bus solutions such as MuleESB and WSO2. We will look at advanced topics such as performance tuning, secure messaging, and the internals of RabbitMQ. Finally we will work through case-studies so that we can see RabbitMQ in action and, if something goes wrong, we'll learn to resolve it in the Troubleshooting section.

Publication date:
December 2015
Publisher
Packt
Pages
262
ISBN
9781783984565

 

Chapter 1. Introducing RabbitMQ

In the world of enterprise messaging systems there are a number of patterns and practices that are already successfully applied in order improve to scalability and interoperability between different components in a system or between varying in size and complexity systems. RabbitMQ is one such messaging solution, which combines powerful messaging capabilities with easy use and availability on a number of target platforms.

The following topics will be covered in this chapter:

  • Fundamentals of enterprise messaging

  • RabbitMQ brief overview

  • RabbitMQ features

  • Comparing RabbitMQ to other technologies

  • Installing RabbitMQ

 

Enterprise messaging


A typical enterprise will have a number of systems that must typically communicate with each other in order to implement a well-defined business process. A question that is frequently tackled for this reason is how to implement the communication channel between these types of systems? For example, consider the following diagram:

The question mark in the preceding picture denotes the communication media for the six systems that are illustrated. In the diagram, we can think of these separate systems as the components of one large system and the problem stays the same. Before discussing the various alternatives for integration, a number of key factors are considered, as follows:

  • Loose coupling: At what degree do the different systems depend on each other or can operate independently?

  • Real-time workload processing: How fast is the communication between the systems?

  • Scalability: How does the entire system scale when more systems are added and the workload demands an increase?

  • Maintainability: How hard it is to maintain the integrated systems?

  • Extensibility: How easy it is to integrate new systems?

Let's assume that the systems communicate directly via some kind of remote procedure calls as shown in the following diagram:

This implies that separate communication links must be established between each pair of systems, which leads to tight coupling, a lot of effort to maintain all of the links, reduced scalability, and reduced extensibility (for every new system that is added, a few more communication links with other systems must be created). However, real-time communication requirements might be met with some additional effort to design the communication links.

A second approach is to use a shared file system in order to exchange files between the systems that are being integrated, as illustrated in the following diagram:

A shared file system is used to provide the communication medium. Each system may export data to a file that can be imported and used by other systems. The fact that each system may support its own data format leads to the fact that each system must have a particular mechanism to import data from every other system that it needs to communicate with. This, on other hand, leads to the same problems that are described in the case of direct communication. Moreover, real-time communication requirements might be more difficult to establish and reading or writing data from disk is also an expensive operation.

A third option is to use a shared database as shown in the following diagram:

Here, all the systems should depend on the same database schema. Although this reduces coupling between systems and improves extensibility (new systems must conform to a single database schema in order to integrate with other systems), real-time workload processing is still an issue. Scalability and maintainability depend directly on the type of database that is being used and they could turn out to be weak factors especially if it is a relational database (this may not be the case if NoSQL solutions are applicable for the particular use case).

Messaging comes to the rescue when considering the problems that arise when applying the previous approaches. Consider the following diagram for the Enterprise Messaging System:

A message is the central unit of communication used in enterprise messaging systems. A message typically consists of the following:

  • A message header: It provides metadata about the message such as encoding, routing information, and security-related information

  • A message body: It provides the actual data that is carried by the message, represented in a proper format

The messaging system itself provides mechanisms to validate, store, route, and transform messages that are being sent between the different systems. Each system is responsible for crafting its own message that is transferred via the messaging system (also called the messaging broker) to other systems that are connected to the broker and configured to receive that particular type of message. Each system may create a message in a proper format that is specified by the protocol of the message broker—meaning that the system is only coupled with that particular protocol. If the broker implements a protocol that is based on a well-recognized standard, then this would further decouple the systems from that particular message broker implementation.

Real-time workload processing is typically quite fast as the particular protocol that is implemented by different messaging brokers is optimized to process message data in a reliable and secure manner with minimal overhead. Most messaging solutions provide a number of facilities that allow easy configuration, management, and monitoring; thus, simplifying maintainability. Clustering support is also considered by most implementations due to scalability and reliability requirements and increasing workload demands. Integrating new systems is a matter of implementing a mechanism for direct communication with the message broker.

In case the different systems provide different implementations of messaging protocols (such as REST, SOAP, JSON-RPC, JMX, AMQP, and many others), a messaging system could further provide various adapters for the different protocols as well as extended mechanisms for routing and transformation of different types of messages—this extended functionality also categorizes the message brokers as Enterprise Service Bus (ESB) solutions. One major drawback of an ESB is that it must implement all the communication requirements of all systems that are being integrated by the ESB, otherwise workarounds must be used in order to implement direct communication between the integrated systems (thus, neglecting the usage of an ESB).

Use cases

There are a variety of scenarios where messaging systems may be applied, such as the following:

  • Financial services: High rate real-time trade transactions handled between different systems

  • Social networking: Activity streams and event propagation between different components in a social network

  • E-mailing: Sending e-mail notifications or digests periodically to a large number of users

  • Processing large volumes of data upon request, such as image rendering

  • Chat services

  • Propagation of events throughout a system

  • Any type of real-time system integration (system of systems)

As you can see, messaging solutions can be applied to a variety of scenarios that typically involve a number of systems that must communicate in a timely manner or perform a large number of time-consuming tasks. Messaging solutions are also extensively being deployed by Cloud providers in order to provide messaging as a service for Cloud-based applications.

Solutions

A wide variety of open source and property messaging solutions are available for use, which are based on the multitude of use cases. The choice of a messaging broker depends on a number of factors, as shown in the following:

  • Supporting tools, documentation and services: These are tools for management and monitoring of the broker along with possible options to receive support, typically the support is guaranteed only for commercial brokers. For open source, this depends on the activeness of the community.

  • Ease of configuration: This shows how easy it is to set up and configure the message broker.

  • Functionality: The features implemented by the solution and their coverage of the usage scenario. Here the supported protocols for message transfer play a key role in the decision.

  • The cost and licensing model.

Patterns

A messaging system provides patterns for communication between system endpoints.

Point-to-point

In a point-to-point communication model, there is exactly one sender and one receiver of a message. In case there are multiple senders that are applicable for the purpose of receiving the message, only one of them succeeds. Such receivers are also referred to as competing consumers, indicating that any of them are eligible to receive the message. The sender does not receive a response in a point-to-point model.

Publish-subscribe

In a Publish-subscribe communication model, there is one sender and multiple receivers (subscribers) of the message. It is a form of fire-and-forget, where the sender does not await for a response once the message is sent to the broker.

Request-response

In a request-response communication model, there is one sender and one receiver that sends a response to the sender of the message.

 

Understanding RabbitMQ


The RabbitMQ messaging server is an open source implementation of the AMQP 0-9-1 protocol (Advanced Message Queuing Protocol) that defines how messages should be queued, routed, and delivered in a reliable and secure manner. AMQP 1.0, which is an OASIS (Organization for the Advancement of Structured Information Standards), is not directly supported in the message broker; however, RabbitMQ provides a plugin for AMQP 1.0 (as it is not backward-compatible with AMQP 0-9-1). OASIS is a non-profit organization that works for the development of a number of technology standards. As an open standard, AMQP promotes interoperability among the messaging brokers that implement the protocol. It also defines the delivery semantics for a message, which dictates how many times that message can be sent from one endpoint to another—zero or once, exactly once or multiple times. As a wire protocol, AMQP provides better performance in regard to other messaging protocols such as XMPP (Extensible Messaging and Presence Protocol).

Before we discuss more about RabbitMQ as a message broker, we will introduce some terminologies from the RabbitMQ world that we will use frequently throughout the book:

  • exchanges: These are the RabbitMQ server endpoints to which the clients will connect and send messages. Each endpoint is identified by a unique key.

  • queues: These are the RabbitMQ server components that buffer messages coming from one or more exchanges and send them to the corresponding message receivers. The messages in a queue can also be offloaded to a persistent storage (such queues are also called durable queues) that provides a higher degree of reliability in case of a failed messaging server; once the server is running again, the messages from persistent storage are placed back in the corresponding queues for transfer to recipients. Each queue is identified by a unique key.

  • bindings: These are the logical link between exchanges and queues. Each binding is a rule that specifies how the exchanges should route messages to queues. A binding may have a routing key that can be used by clients in order to specify the routing semantics of a message.

  • virtual hosts: The logical units that divide RabbitMQ server components (such as exchanges, queues, and users) into separate groups for better administration and access control. Each AMQP client connection is bound to a concrete virtual host.

The AMQP protocol allows a client to establish a one-way logical link to send messages for exchange. Each logical link is a separate AMQP channel that may provide additional options for the reliable transfer of messages. In this regard, a single-client TCP connection to the RabbitMQ server allows multiple AMQP channels of communication. Since AMQP does not provide the capability to retrieve the list of queues, exchanges, or bindings that are defined in the RabbitMQ message broker, client applications must specify the exchange name, queue names, and, optionally, routing information by means of routing keys for particular bindings. AMQP is a programmatic protocol that allows its clients to create and delete exchanges, queues, and bindings when necessary. RabbitMQ addresses some limitations of AMQP by providing custom extensions apart from the fact that the AMQP protocol itself is extensible. In order to simply application development, RabbitMQ provides several exchange types out of the box, as follows:

  • direct exchange: This delivers a message based on a routing key that is provided in the message header (bindings should already be defined between the direct exchange and the queue). There is a pre-created direct exchange with the name .amq.direct. A specialized type of a direct exchange called default exchange with the empty string as the exchange name is also pre-created in the message broker. It has the special property where the binding key that is specified by the client should match the name of the queue to which a message is routed.

  • fanout exchange: This delivers a message to all the queues that are bound to the exchange; it can be used to establish a broadcast mechanism for the delivery of messages to the queues. There is a pre-created fanout exchange with name .amq.fanout.

  • topic exchange: This delivers the message to queues based on a routing filter specified between the topic exchange and queues; it can be used to establish a multicast mechanism for the delivery of messages. There is a pre-created topic exchange with the name .amq.topic.

  • headers exchange: This can be used to deliver messages to queues based on other message header attributes (and not the routing key). There are two pre-created headers exchanges with names .amq.headers and .amq.match.

Receivers can either subscribe to a queue in order to receive messages (also called push-style communication) or request messages on demand from a queue (also called pull-style communication).

Features

The RabbitMQ message broker provides a number of features and tools that support production-level deployment, management, and configuration of the RabbitMQ server instances as shown in the following:

  • support for multiple protocols: Apart from AMQP, RabbitMQ provides support for the STOMP, MQTT, and HTTP protocols by the means of RabbitMQ plug-ins.

  • routing capabilities: As we already saw, we can implement rules to route messages between exchanges and queues by means of bindings, moreover, more custom exchange types can be defined that can provide additional routing capabilities.

  • support for multiple programming languages: There are a variety of supported clients for a great variety of programming languages.

  • reliable delivery: This is a mechanism that guarantees successful message delivery by the means of acknowledgements. It can be enabled between the producer and the broker as well as the broker and the consumer.

  • clustering: This provides a mechanism to implement scalable applications in terms of the RabbitMQ message broker.

  • federation: This is an alternative mechanism to implement scalable applications with RabbitMQ by the means of transferring messages between exchanges and queues in different broker instances without the need to create a RabbitMQ cluster.

  • high availability: This ensures that if a broker fails, communication will be redirected to a different broker instance. It is implemented by the means of mirroring queues; messages from a queue on a master broker instance are copied to a queue on a slave broker instance and, once the message is acknowledge, the messages are discarded from both the master and slave instances.

  • management and monitoring: A number of utilities are built around the RabbitMQ broker server that provide these capabilities.

  • Authentication and access control.

  • pluggable architecture: RabbitMQ provides a mechanism to extend its functionality by the means of RabbitMQ plug-ins.

All of these features will be covered in detail in the next chapters.

Comparison with other technologies

As RabbitMQ is not the only player in the world of enterprise messaging solutions, it is good to see what makes RabbitMQ different compared to other messaging systems. A short list of alternative solutions (some of them also implementing the AMQP protocol) may include systems such as Apache ActiveMQ, Apache Kafka, Apache Qpid, JBoss Messaging, Microsoft BizTalk Server, and WebSphere Message Broker. There are different benchmarks that can be found throughout the internet that show us the relative results in comparison to the different types of brokers in terms of message sending (from publisher to broker) and message delivery (from broker to consumer). In case you need to compare RabbitMQ with the previously mentioned or other messaging solutions, you can apply the following strategy:

  • Select a subset of technologies that are suitable for your use case based on the variety of factors that are listed at the beginning of this chapter

  • Perform different types of benchmark based on the variations of size and number of messages that will be sent for the purpose of processing by each solution in the comparison group, based on the format of messages for the particular use case

Installation

You can download a RabbitMQ distribution package for the operating system of your choice from http://www.rabbitmq.com/Windows.

For Windows operating systems, you have the ability to use the provided RabbitMQ installer (the simpler alternative) or manually install the broker using the provided zip distribution archive (requires additional setup of Windows system paths). We will provide an overview of the installation process for Windows 7 using the installer for Rabbit 3.3.5 (rabbitmq-server-3.3.5.exe) that is quite straight-forward. Initially, the installer checks whether Erlang is installed on the target Windows system and, if it cannot find it, a dialog prompts you to install it, as shown in the following image:

If you click Yes, the dialog redirects you to the official Erlang website. There, you will find the appropriate binaries for your 32-bit/64-bit Windows operating system. Download and install the Erlang 17.3 distribution (compatible with RabbitMQ 3.3.5) for 64-bit Windows (otp_win64_17.3.exe):

File associations can be established and the Erlang documentation can be installed as a part of the Erlang installation process:

The next step is to specify the location to install Erlang:

Finally, you have the option to place an Erlang shortcut in the Start menu folder. After the installation is finished, you can run the RabbitMQ 3.3.5 server installer again:

You can specify in addition that Start menu items and a Windows service should be added along with RabbitMQ server installation. Adding a Windows service for RabbitMQ server is usually recommended as it provides a convenient mechanism to manage a RabbitMQ server instance:

The final step is to specify the location to install the RabbitMQ server. Once the installation is complete, the installer tries to start the RabbitMQ server and, if your Windows firewall is turned on, you might be prompted to allow access to the RabbitMQ server in order to open a port on the target machine (the default port is 5672 for a RabbitMQ server node instance). In order to check whether the RabbitMQ service is running, you can open services.msc from the Windows Run menu and check whether the RabbitMQ service has started. Additionally, you can check whether the RabbitMQ instance node is initiated, by default on port 5672, by executing from the command prompt:

netstat –a

RabbitMQ installation also provides a number of command-line utilities that you can use in order to manage the RabbitMQ instance, which is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory. You can use the rabbitmqctl utility to check the status of a broker, start, or stop it:

rabbitmqctl.bat status
rabbitmqctl.bat stop
rabbitmqctl.bat start

In addition to this, rabbitmqctl provides a number of other commands that can be used to manage the RabbitMQ broker. There is a RabbitMQ management plugin that provides the ability to manage a RabbitMQ broker from a web-based interface and in particular to do the following:

  • Manage broker objects such as message queues, users, and permissions

  • Send messages to the broker

  • Receive messages from the broker

  • Monitor and manage connections to the broker

  • Monitor broker workload

  • Monitor resource usage such as memory, processes, and file descriptors that are used by the broker

These are included in the set of plugins that are installed by default; however, it must be enabled by executing the rabbitmq-plugins utility that is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory from the following command prompt:

rabbitmq-plugins.bat enable rabbitmq_management

After the management plugin is enabled, you have to restart the RabbitMQ server:

rabbitmq-server.bat restart

The management plugin starts an http server, on port 15672, by default in order to verify that the plugin is trying to open http://localhost:15672/ from a browser. You will be prompted to provide a Username and Password in order to login:

By default, RabbitMQ installs with a user with a guest name and guest password that are only available from local host connections to the broker. In the next chapter, we will see how to manage users for a RabbitMQ server.

Linux

For various Linux distributions, there are out-of-the-box packages provided for the RabbitMQ server. Some Linux distributions also provide you with the ability to install the broker directly from a package repository. We will provide an overview of the installation process for Ubuntu 12.04 Desktop edition based on a package repository that we can also download and install directly, a RabbitMQ Debian package for the purpose. To install the broker and enable the management plugin, open a terminal and execute the following command:

echo "deb http://www.rabbitmq.com/debian/ testing main"  | sudo tee  /etc/apt/sources.list.d/rabbitmq.list > /dev/null
sudo wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update
sudo apt-get install rabbitmq-server –y
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart

Tip

Downloading the example code

You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The server installation also installs utilities for the management of the RabbitMQ server—used in the same way as in the Windows installation of RabbitMQ.

 

Case study: CSN (Corporate Social Network)


The Corporate Social Network (CSN) is a social networking service that is being deployed in an enterprise and allows its users to upload content and interact with each other. In particular, the system allows the user to post blogs, upload files, subscribe to other user profiles (in order to track the activity of other users), and chat with other users. The social network uses RabbitMQ in order to process events that have been triggered by user activity, trigger long-running jobs (such as batch file uploading), and serve as a backbone for the delivery of chat messages from the chat feature of the social network. The following diagram provides a high-level overview of the components of the system:

We will design the system from the very beginning and then start expanding it. In the meantime, we will demonstrate the various capabilities of RabbitMQ by applying them to the extensions of the social network.

 

Summary


In this chapter, we covered the fundamentals of enterprise messaging solutions and discussed the features of RabbitMQ along with the installation process. A brief comparison with other messaging brokers was provided in order to reveal what the strengths and weaknesses of RabbitMQ are compared to the other alternatives. We also introduced a case study project CSN that makes use of RabbitMQ as a messaging solution for propagation of events throughout the system and lays the basis for further demonstrations on the various features of RabbitMQ.

 

Exercises


Attempt the following questions:

  1. What is messaging?

  2. What are the typical components of a message broker?

  3. What appropriate usage scenarios can you think of for the application of messaging systems?

  4. Which message patterns does RabbitMQ support and how?

  5. What are the advantages of using AMQP for messaging compared to other protocols?

  6. What are the different features that are supported by RabbitMQ?

  7. What are the prerequisites for RabbitMQ installation on a target operating system?

About the Author

  • Martin Toshev

    Martin Toshev is an IT consultant, Java enthusiast, and one of the leads of the Bulgarian Java Users group (BG JUG). He is a certified Java professional (SCJP6) and a certified IBM cloud computing solution advisor. His areas of interest include the wide range of Java-related technologies (such as Servlets, JSP, JAXB, JAXP, JMS, JMX, JAX-RS, JAX-WS, Hibernate, Spring Framework, Liferay Portal, and Eclipse RCP), cloud computing technologies, cloud-based software architectures, enterprise application integration, and relational and NoSQL databases.

    You can reach him for any Java and FOSS-related topics (especially Eclipse and the OpenJDK). Martin is a regular speaker at Java conferences and one of the organizers behind the jPrime conference in Bulgaria (http://jprime.io/).

    Browse publications by this author

Latest Reviews

(3 reviews total)
Good
My delivery was delayed a bit (due to local post) but finally got it.
Great book guide written in clear and concise format for anyone who wants to get into messaging and rabbitMQ setup and use.
Book Title
Unlock this book and the full library for FREE
Start free trial