Learning RabbitMQ

Chapter 1. Introducing RabbitMQ

In the world of enterprise messaging systems there are a number of patterns and practices that are already successfully applied in order improve to scalability and interoperability between different components in a system or between varying in size and complexity systems. RabbitMQ is one such messaging solution, which combines powerful messaging capabilities with easy use and availability on a number of target platforms.

The following topics will be covered in this chapter:

Fundamentals of enterprise messaging
RabbitMQ brief overview
RabbitMQ features
Comparing RabbitMQ to other technologies
Installing RabbitMQ

Enterprise messaging

A typical enterprise will have a number of systems that must typically communicate with each other in order to implement a well-defined business process. A question that is frequently tackled for this reason is how to implement the communication channel between these types of systems? For example, consider the following diagram:

The question mark in the preceding picture denotes the communication media for the six systems that are illustrated. In the diagram, we can think of these separate systems as the components of one large system and the problem stays the same. Before discussing the various alternatives for integration, a number of key factors are considered, as follows:

Loose coupling: At what degree do the different systems depend on each other or can operate independently?
Real-time workload processing: How fast is the communication between the systems?
Scalability: How does the entire system scale when more systems are added and the workload demands an increase?
Maintainability: How hard it is to maintain the integrated systems?
Extensibility: How easy it is to integrate new systems?

Let's assume that the systems communicate directly via some kind of remote procedure calls as shown in the following diagram:

This implies that separate communication links must be established between each pair of systems, which leads to tight coupling, a lot of effort to maintain all of the links, reduced scalability, and reduced extensibility (for every new system that is added, a few more communication links with other systems must be created). However, real-time communication requirements might be met with some additional effort to design the communication links.

A second approach is to use a shared file system in order to exchange files between the systems that are being integrated, as illustrated in the following diagram:

A shared file system is used to provide the communication medium. Each system may export data to a file that can be imported and used by other systems. The fact that each system may support its own data format leads to the fact that each system must have a particular mechanism to import data from every other system that it needs to communicate with. This, on other hand, leads to the same problems that are described in the case of direct communication. Moreover, real-time communication requirements might be more difficult to establish and reading or writing data from disk is also an expensive operation.

A third option is to use a shared database as shown in the following diagram:

Here, all the systems should depend on the same database schema. Although this reduces coupling between systems and improves extensibility (new systems must conform to a single database schema in order to integrate with other systems), real-time workload processing is still an issue. Scalability and maintainability depend directly on the type of database that is being used and they could turn out to be weak factors especially if it is a relational database (this may not be the case if NoSQL solutions are applicable for the particular use case).

Messaging comes to the rescue when considering the problems that arise when applying the previous approaches. Consider the following diagram for the Enterprise Messaging System:

A message is the central unit of communication used in enterprise messaging systems. A message typically consists of the following:

A message header: It provides metadata about the message such as encoding, routing information, and security-related information
A message body: It provides the actual data that is carried by the message, represented in a proper format

The messaging system itself provides mechanisms to validate, store, route, and transform messages that are being sent between the different systems. Each system is responsible for crafting its own message that is transferred via the messaging system (also called the messaging broker) to other systems that are connected to the broker and configured to receive that particular type of message. Each system may create a message in a proper format that is specified by the protocol of the message broker—meaning that the system is only coupled with that particular protocol. If the broker implements a protocol that is based on a well-recognized standard, then this would further decouple the systems from that particular message broker implementation.

Real-time workload processing is typically quite fast as the particular protocol that is implemented by different messaging brokers is optimized to process message data in a reliable and secure manner with minimal overhead. Most messaging solutions provide a number of facilities that allow easy configuration, management, and monitoring; thus, simplifying maintainability. Clustering support is also considered by most implementations due to scalability and reliability requirements and increasing workload demands. Integrating new systems is a matter of implementing a mechanism for direct communication with the message broker.

In case the different systems provide different implementations of messaging protocols (such as REST, SOAP, JSON-RPC, JMX, AMQP, and many others), a messaging system could further provide various adapters for the different protocols as well as extended mechanisms for routing and transformation of different types of messages—this extended functionality also categorizes the message brokers as Enterprise Service Bus (ESB) solutions. One major drawback of an ESB is that it must implement all the communication requirements of all systems that are being integrated by the ESB, otherwise workarounds must be used in order to implement direct communication between the integrated systems (thus, neglecting the usage of an ESB).

Use cases

There are a variety of scenarios where messaging systems may be applied, such as the following:

Financial services: High rate real-time trade transactions handled between different systems
Social networking: Activity streams and event propagation between different components in a social network
E-mailing: Sending e-mail notifications or digests periodically to a large number of users
Processing large volumes of data upon request, such as image rendering
Chat services
Propagation of events throughout a system
Any type of real-time system integration (system of systems)

As you can see, messaging solutions can be applied to a variety of scenarios that typically involve a number of systems that must communicate in a timely manner or perform a large number of time-consuming tasks. Messaging solutions are also extensively being deployed by Cloud providers in order to provide messaging as a service for Cloud-based applications.

Solutions

A wide variety of open source and property messaging solutions are available for use, which are based on the multitude of use cases. The choice of a messaging broker depends on a number of factors, as shown in the following:

Supporting tools, documentation and services: These are tools for management and monitoring of the broker along with possible options to receive support, typically the support is guaranteed only for commercial brokers. For open source, this depends on the activeness of the community.
Ease of configuration: This shows how easy it is to set up and configure the message broker.
Functionality: The features implemented by the solution and their coverage of the usage scenario. Here the supported protocols for message transfer play a key role in the decision.
The cost and licensing model.

Patterns

A messaging system provides patterns for communication between system endpoints.

Point-to-point

In a point-to-point communication model, there is exactly one sender and one receiver of a message. In case there are multiple senders that are applicable for the purpose of receiving the message, only one of them succeeds. Such receivers are also referred to as competing consumers, indicating that any of them are eligible to receive the message. The sender does not receive a response in a point-to-point model.

Publish-subscribe

In a Publish-subscribe communication model, there is one sender and multiple receivers (subscribers) of the message. It is a form of fire-and-forget, where the sender does not await for a response once the message is sent to the broker.

Request-response

In a request-response communication model, there is one sender and one receiver that sends a response to the sender of the message.

Understanding RabbitMQ

The RabbitMQ messaging server is an open source implementation of the AMQP 0-9-1 protocol (Advanced Message Queuing Protocol) that defines how messages should be queued, routed, and delivered in a reliable and secure manner. AMQP 1.0, which is an OASIS (Organization for the Advancement of Structured Information Standards), is not directly supported in the message broker; however, RabbitMQ provides a plugin for AMQP 1.0 (as it is not backward-compatible with AMQP 0-9-1). OASIS is a non-profit organization that works for the development of a number of technology standards. As an open standard, AMQP promotes interoperability among the messaging brokers that implement the protocol. It also defines the delivery semantics for a message, which dictates how many times that message can be sent from one endpoint to another—zero or once, exactly once or multiple times. As a wire protocol, AMQP provides better performance in regard to other messaging protocols such as XMPP (Extensible Messaging and Presence Protocol).

Before we discuss more about RabbitMQ as a message broker, we will introduce some terminologies from the RabbitMQ world that we will use frequently throughout the book:

exchanges: These are the RabbitMQ server endpoints to which the clients will connect and send messages. Each endpoint is identified by a unique key.
queues: These are the RabbitMQ server components that buffer messages coming from one or more exchanges and send them to the corresponding message receivers. The messages in a queue can also be offloaded to a persistent storage (such queues are also called durable queues) that provides a higher degree of reliability in case of a failed messaging server; once the server is running again, the messages from persistent storage are placed back in the corresponding queues for transfer to recipients. Each queue is identified by a unique key.
bindings: These are the logical link between exchanges and queues. Each binding is a rule that specifies how the exchanges should route messages to queues. A binding may have a routing key that can be used by clients in order to specify the routing semantics of a message.
virtual hosts: The logical units that divide RabbitMQ server components (such as exchanges, queues, and users) into separate groups for better administration and access control. Each AMQP client connection is bound to a concrete virtual host.

The AMQP protocol allows a client to establish a one-way logical link to send messages for exchange. Each logical link is a separate AMQP channel that may provide additional options for the reliable transfer of messages. In this regard, a single-client TCP connection to the RabbitMQ server allows multiple AMQP channels of communication. Since AMQP does not provide the capability to retrieve the list of queues, exchanges, or bindings that are defined in the RabbitMQ message broker, client applications must specify the exchange name, queue names, and, optionally, routing information by means of routing keys for particular bindings. AMQP is a programmatic protocol that allows its clients to create and delete exchanges, queues, and bindings when necessary. RabbitMQ addresses some limitations of AMQP by providing custom extensions apart from the fact that the AMQP protocol itself is extensible. In order to simply application development, RabbitMQ provides several exchange types out of the box, as follows:

direct exchange: This delivers a message based on a routing key that is provided in the message header (bindings should already be defined between the direct exchange and the queue). There is a pre-created direct exchange with the name .amq.direct. A specialized type of a direct exchange called default exchange with the empty string as the exchange name is also pre-created in the message broker. It has the special property where the binding key that is specified by the client should match the name of the queue to which a message is routed.
fanout exchange: This delivers a message to all the queues that are bound to the exchange; it can be used to establish a broadcast mechanism for the delivery of messages to the queues. There is a pre-created fanout exchange with name .amq.fanout.
topic exchange: This delivers the message to queues based on a routing filter specified between the topic exchange and queues; it can be used to establish a multicast mechanism for the delivery of messages. There is a pre-created topic exchange with the name .amq.topic.
headers exchange: This can be used to deliver messages to queues based on other message header attributes (and not the routing key). There are two pre-created headers exchanges with names .amq.headers and .amq.match.

Receivers can either subscribe to a queue in order to receive messages (also called push-style communication) or request messages on demand from a queue (also called pull-style communication).

Features

The RabbitMQ message broker provides a number of features and tools that support production-level deployment, management, and configuration of the RabbitMQ server instances as shown in the following:

support for multiple protocols: Apart from AMQP, RabbitMQ provides support for the STOMP, MQTT, and HTTP protocols by the means of RabbitMQ plug-ins.
routing capabilities: As we already saw, we can implement rules to route messages between exchanges and queues by means of bindings, moreover, more custom exchange types can be defined that can provide additional routing capabilities.
support for multiple programming languages: There are a variety of supported clients for a great variety of programming languages.
reliable delivery: This is a mechanism that guarantees successful message delivery by the means of acknowledgements. It can be enabled between the producer and the broker as well as the broker and the consumer.
clustering: This provides a mechanism to implement scalable applications in terms of the RabbitMQ message broker.
federation: This is an alternative mechanism to implement scalable applications with RabbitMQ by the means of transferring messages between exchanges and queues in different broker instances without the need to create a RabbitMQ cluster.
high availability: This ensures that if a broker fails, communication will be redirected to a different broker instance. It is implemented by the means of mirroring queues; messages from a queue on a master broker instance are copied to a queue on a slave broker instance and, once the message is acknowledge, the messages are discarded from both the master and slave instances.
management and monitoring: A number of utilities are built around the RabbitMQ broker server that provide these capabilities.
Authentication and access control.
pluggable architecture: RabbitMQ provides a mechanism to extend its functionality by the means of RabbitMQ plug-ins.

All of these features will be covered in detail in the next chapters.

Comparison with other technologies

As RabbitMQ is not the only player in the world of enterprise messaging solutions, it is good to see what makes RabbitMQ different compared to other messaging systems. A short list of alternative solutions (some of them also implementing the AMQP protocol) may include systems such as Apache ActiveMQ, Apache Kafka, Apache Qpid, JBoss Messaging, Microsoft BizTalk Server, and WebSphere Message Broker. There are different benchmarks that can be found throughout the internet that show us the relative results in comparison to the different types of brokers in terms of message sending (from publisher to broker) and message delivery (from broker to consumer). In case you need to compare RabbitMQ with the previously mentioned or other messaging solutions, you can apply the following strategy:

Select a subset of technologies that are suitable for your use case based on the variety of factors that are listed at the beginning of this chapter
Perform different types of benchmark based on the variations of size and number of messages that will be sent for the purpose of processing by each solution in the comparison group, based on the format of messages for the particular use case

Installation

You can download a RabbitMQ distribution package for the operating system of your choice from http://www.rabbitmq.com/Windows.

For Windows operating systems, you have the ability to use the provided RabbitMQ installer (the simpler alternative) or manually install the broker using the provided zip distribution archive (requires additional setup of Windows system paths). We will provide an overview of the installation process for Windows 7 using the installer for Rabbit 3.3.5 (rabbitmq-server-3.3.5.exe) that is quite straight-forward. Initially, the installer checks whether Erlang is installed on the target Windows system and, if it cannot find it, a dialog prompts you to install it, as shown in the following image:

If you click Yes, the dialog redirects you to the official Erlang website. There, you will find the appropriate binaries for your 32-bit/64-bit Windows operating system. Download and install the Erlang 17.3 distribution (compatible with RabbitMQ 3.3.5) for 64-bit Windows (otp_win64_17.3.exe):

File associations can be established and the Erlang documentation can be installed as a part of the Erlang installation process:

The next step is to specify the location to install Erlang:

Finally, you have the option to place an Erlang shortcut in the Start menu folder. After the installation is finished, you can run the RabbitMQ 3.3.5 server installer again:

You can specify in addition that Start menu items and a Windows service should be added along with RabbitMQ server installation. Adding a Windows service for RabbitMQ server is usually recommended as it provides a convenient mechanism to manage a RabbitMQ server instance:

The final step is to specify the location to install the RabbitMQ server. Once the installation is complete, the installer tries to start the RabbitMQ server and, if your Windows firewall is turned on, you might be prompted to allow access to the RabbitMQ server in order to open a port on the target machine (the default port is 5672 for a RabbitMQ server node instance). In order to check whether the RabbitMQ service is running, you can open services.msc from the Windows Run menu and check whether the RabbitMQ service has started. Additionally, you can check whether the RabbitMQ instance node is initiated, by default on port 5672, by executing from the command prompt:

netstat –a

RabbitMQ installation also provides a number of command-line utilities that you can use in order to manage the RabbitMQ instance, which is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory. You can use the rabbitmqctl utility to check the status of a broker, start, or stop it:

rabbitmqctl.bat status
rabbitmqctl.bat stop
rabbitmqctl.bat start

In addition to this, rabbitmqctl provides a number of other commands that can be used to manage the RabbitMQ broker. There is a RabbitMQ management plugin that provides the ability to manage a RabbitMQ broker from a web-based interface and in particular to do the following:

Manage broker objects such as message queues, users, and permissions
Send messages to the broker
Receive messages from the broker
Monitor and manage connections to the broker
Monitor broker workload
Monitor resource usage such as memory, processes, and file descriptors that are used by the broker

These are included in the set of plugins that are installed by default; however, it must be enabled by executing the rabbitmq-plugins utility that is located under the rabbitmq_server-3.3.5\sbin folder in the RabbitMQ installation directory from the following command prompt:

rabbitmq-plugins.bat enable rabbitmq_management

After the management plugin is enabled, you have to restart the RabbitMQ server:

rabbitmq-server.bat restart

The management plugin starts an http server, on port 15672, by default in order to verify that the plugin is trying to open http://localhost:15672/ from a browser. You will be prompted to provide a Username and Password in order to login:

By default, RabbitMQ installs with a user with a guest name and guest password that are only available from local host connections to the broker. In the next chapter, we will see how to manage users for a RabbitMQ server.

Linux

For various Linux distributions, there are out-of-the-box packages provided for the RabbitMQ server. Some Linux distributions also provide you with the ability to install the broker directly from a package repository. We will provide an overview of the installation process for Ubuntu 12.04 Desktop edition based on a package repository that we can also download and install directly, a RabbitMQ Debian package for the purpose. To install the broker and enable the management plugin, open a terminal and execute the following command:

echo "deb http://www.rabbitmq.com/debian/ testing main"  | sudo tee  /etc/apt/sources.list.d/rabbitmq.list > /dev/null
sudo wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update
sudo apt-get install rabbitmq-server –y
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart

Tip

Downloading the example code

You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The server installation also installs utilities for the management of the RabbitMQ server—used in the same way as in the Windows installation of RabbitMQ.

John Jan 25, 2016

This book is definitely a winner - the only reason that I cannot give it 5-stars is that there are still some polish points that needs to be added in a few places, but once that is fixed up I'll be moving it to 5 stars - the content and overview of RabbitMQ presented in this book is very, very good. So, how did I wind up with this book and why do you care what I think? I am a midrange automation guy, and I work for a global IT organization - so is see and read a *lot* of documentation and I write a fair bit of my own. I also have a love for well-engineered products and for open-source. I found out that OpenStack uses RabbitMQ - a major point for it right there - as its internal messaging system last year during an End-to-end mid range automation design I was writing. I decided that I wanted to know more, so I got a copy for Christmas from my family. Now that you believe I'm trustworthy, here's what I liked about the book, and why I think that you'll like it if you want to get a really good understanding of RabbitMQ here.1. The author goes from the most basic of ME concepts and smoothly builds all the way to discussing and debugging the innards of the system in Erlang.2. I never felt lost as I read through the book - that's pretty hard to pull off. My congratulations to the author!2. There are lots of code examples in Java. Now I'm a perl/shell guy and I don't much like Java, but. I understood these quite easily.3. I understand a good deal of what makes RabbitMQ different.4. Very good breath covering many associated but important topics, ones you usually have to buy other books to learn.- LDAP integration.- Securing each component of RabbitMQ.- An assessment of generic weaknesses in all RabbitMQ with appropriate counter-measures.- Integration with multiple ESB systems. This one really went above and beyond, so thank you for bringing this one in!- Clustering/HA and both together.- Performance tuning and monitoring.- Even how to gent involved in contributing to RabbitMQ development!5. While every demo is applicable to the corporate world, the author sets all of them up so you can experiment on just one (1) box at home and get a full set of practice in every topic. Since I'm a practical, hands-on person who needs to play with the tech to learn it, this has been a fantastic for me to be able to play with RabbitMQ without needing a corporate network and a full lab in order to follow through with the examples and get some experience with RabbitMQ.You normally don't get that kind of an education from a "Learning" book. I'm duly impressed and glad that I read this one first!So, in summary I would definitely recommend this book, and now I'll be looking to put RabbitMQ to greater ues - and also other books from this author. The author managed to convince me: congratulations, Mr. Toshev, you've made me into a RabbitMQ convert.Happy reading everyone!

Amazon Verified review

White Knight Jan 26, 2016

Pretty good, could do on more about enterprise architecture patterns and best practices.

Rick O'Shay Jun 10, 2016

A poorly written, error riddled regurgitation of information scraped and paraphrased from other sources with no attribution -- seems to be a theme with this publisher. I don't think it's a stretch to say this is the worst blog scraping in paperback form ever assembled on the topic.

Learning RabbitMQ: Build and optimize efficient messaging applications with ease

What do you get with Print?

Learning RabbitMQ

Chapter 1. Introducing RabbitMQ

Enterprise messaging

Use cases

Solutions

Patterns

Point-to-point

Publish-subscribe

Request-response

Understanding RabbitMQ

Features

Comparison with other technologies

Installation

Linux

Tip

Case study: CSN (Corporate Social Network)

Summary

Exercises

Page 1 of 6

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

Learning RabbitMQ: Build and optimize efficient messaging applications with ease

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Use cases

Solutions

Patterns

Point-to-point

Publish-subscribe

Request-response

Linux

Key benefits

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access