Mastering RabbitMQ

3.8 (4 reviews total)
By Emrah Ayanoglu , Yusuf Aytaş , Dotan Nahum
  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Getting Started

About this book

RabbitMQ is one of the most powerful Open Source message broker software, which is widely used in tech companies such as Mozilla, VMware, Google, AT&T, and so on. RabbitMQ gives you lots of fantastic and easy-to-manage functionalities to control and manage the messaging facility with lots of community support. As scalability is one of our major modern problems, messaging with RabbitMQ is the main part of the solution to this problem.

This book explains and demonstrates the RabbitMQ server in a detailed way. It provides you with lots of real-world examples and advanced solutions to tackle the scalability issues.

You’ll begin your journey with the installation and configuration of the RabbitMQ server, while also being given specific details pertaining to the subject. Next, you’ll study the major problems that our server faces, including scalability and high availability, and try to get the solutions for both of these issues by using the RabbitMQ mechanisms. Following on from this, you’ll get to design and develop your own plugins using the Erlang language and RabbitMQ’s internal API. This knowledge will help you to start with the management and monitoring of the messages, tools, and applications. You’ll also gain an understanding of the security and integrity of the messaging facilities that RabbitMQ provides. In the last few chapters, you will build and keep track of your clients (senders and receivers) using Java, Python, and C#.

Publication date:
January 2016
Publisher
Packt
Pages
286
ISBN
9781783981526

 

Chapter 1. Getting Started

Scalability is one of the major problems of our time, and messaging is an integral part of the solution. It finally comes down to the message broker software to manage and control messaging between applications, processes, and threads. Message brokers can help to solve scalability issues and architectural issues, such as coupling.

RabbitMQ is one of the most powerful open source message broker software that is widely used in the tech companies such as Mozilla, VMware, Google, AT&T, and so on. RabbitMQ is a highly configurable messaging platform developed and supported by a knowledgeable and committed community.

Before diving into the details and technologies behind the RabbitMQ, let's introduce you to the topics that we will cover in this chapter:

  • A brief introduction to message brokers and the message queue

  • An introduction to advanced message queue protocol

  • Getting started with RabbitMQ

  • Installing RabbitMQ

  • Starting RabbitMQ

  • Summary

 

Message brokers and message queue


Recently, software systems evolved dramatically. Applications have to communicate with other applications, these applications can be internal and external to the application itself. For the same application, we may have different type of clients, such as browsers, mobile clients, and so on. Hence, we absolutely need a communication layer between internal applications and between applications and clients. We need to deliver different messages to different applications or clients. Delivering messages can be a bottleneck if the communication layer isn't scalable. Pursuing scalable systems for communication layer leads us to Message Brokers and Message Queues. Let's now discuss what Message Brokers and Message Queues are.

Message brokers

A Message Broker is an architectural pattern that can receive messages from multiple destinations, determine the correct destination, and route the message along the correct route, as stated in the book Enterprise Integration Patterns by Hohpe and Woolf. Message brokers enable systems to deal with messaging and routing by mediating communication among components. Once applications implement a message broker pattern, it decreases the coupling between application components.

Message Brokers are centralized, in the architectural sense, to control and manage all messages. Therefore, all of the incoming and outgoing messages are sent through Message Brokers, which analyze and deliver the messages to their correct destination. This procedural step can be understood with the following diagram:

Message Broker

Message Brokers address the following concerns in the communication layer:

  • Transforming messages to alternative formats

  • Routing messages to destinations

  • Supporting different types of patterns to send messages

  • Receiving and responding to events

  • Performing message aggregation

  • Persisting the message states

  • Ensuring the receiving and sending of message

  • Decoupling the destination software systems

Many tasks of the Message Broker need a Message Queue for exchanging or passing data to the destination. The next section covers Message Queues. We will talk about the mechanism behind Message Brokers in Chapter 3, Architecture and Messaging.

Message Queues

A Message Queue is, briefly, a queue for messaging. Queue is the basic data structure behind the functioning of a Message Queue. Message Queue operations are similar to Queue data structure operations, such as the enqueue and dequeu operations. An enqueue operation leads to adding an element to the back of the queue. A dequeue operation leads to the deletion of an element from the front of the queue.

Message Queues provide concurrent and asynchronous operations to scale applications. In a message queue, messages wait up until a message is retrieved by an application. Let's take a look at the following diagram:

Message Queue

Different types of standards and protocols define the Message Queuing specifications. Some protocols are open to everyone; however, some protocols are closed. Let's come back to our topic. RabbitMQ uses Advanced Message Queuing Protocol (AMQP) that determines the policies of the Message Queues. The next topic will cover detailed information on AMQP. Chapter 3, Architecture and Messaging, covers the detailed explanation of Message Queues.

 

An introduction to the advanced message queue protocol


John O'Hara from J. P. Morgan started AMQP in 2003. He put incredible amount of work into it. Then, J. P. Morgan approached other firms to establish an organization for creating open standards in messaging. According to AMOP's official website (http://www.amqp.org), AMQP is an open standard for passing messages between applications or organizations. So, AMQP just defines the messaging properties, queue properties, how messages are routed between applications and clients, how Message Brokers ensure that the message is received or sent, and other concerns such as reliability and security.

According to the AMQP website (http://www.amqp.org), AMQP has lots of capabilities to accomplish goals:

  • Security

  • Reliability

  • Interoperability

  • Standard

  • Open standard

Interoperability and reliability are very important for today's software engineering problems. The power of AMQP comes from its features like interoperability, reliability and so forth. Especially, with interoperability, we can use different types of technologies in sender and receiver. The main problem for most of the Internet giants is scalability. Scalability has direct relationship to reliability. Chapter 3, Architecture and Messaging, covers the details and specifications of AMQP.

 

An overview of RabbitMQ


As mentioned earlier, RabbitMQ is an open source Message Broker software that tries to solve messaging problems by implementing the AMQP. RabbitMQ is licensed with Mozilla Public License. RabbitMQ became part of GoPivotal in May 2013 and the community has helped in the development of RabbitMQ. Since then, the community has been trying to improve RabbitMQ.

RabbitMQ logo

As we explained, RabbitMQ solves messaging problems; however, you may ask yourself what kind of messaging problems RabbitMQ solves. In summary, RabbitMQ has the following functionalities to solve messaging problems:

  • Ensures that messages are sent and received

  • Routes the messages to the correct destinations

  • Saves the state of the messages

  • Supports multiple transportation protocols (AMQP, MQTT, STOMP, HTTP)

  • Supports clustering

  • Highly scalable, reliable and available

  • Extendible with plugins

  • Supports clients for almost any of language

  • Large community support also provides commercial support

Now that we have discussed the concepts behind the RabbitMQ, let's try to install and use RabbitMQ on your computer.

 

Installation of RabbitMQ


Installation of RabbitMQ is not distinctly different from other software in different operation systems. Unix-based operating systems can build RabbitMQ from source code and Microsoft Windows can run the standard MSI installers. RabbitMQ installation files can be found in the download webpage of the RabbitMQ website, as shown in the following image:

Download Webpage of RabbitMQ

The only prerequisite for the RabbitMQ installation is the Erlang runtime environment because RabbitMQ runs on the Erlang VM. Therefore, we have to install Erlang before installing the RabbitMQ. Erlang can be downloaded from the Erlang download webpage, as shown in the following image, and installation instructions will be covered in the topics that follow:

Download Webpage of Erlang

Now, let's to talk about the installation of both RabbitMQ and Erlang on Windows, Mac OS X, Ubuntu, Fedora, and Amazon Web Services.

Windows

RabbitMQ runs on both 32-bit and 64-bit machines from the same package. Erlang is installed either as 32-bit and 64-bit. So, RabbitMQ can be easily installed on the Windows operating system. Let's install these stuff for running RabbitMQ.

Firstly, we should install the Erlang runtime environment on Windows. Erlang has Windows installers for 32-bit and 64-bit as shown in the previous image. We can easily download the related binary file to our computer and install Erlang using it:

Installation of Erlang in Windows

After installing the Erlang runtime environment, we've completed the requirements of RabbitMQ installation. The next step is to download and install the RabbitMQ binary file with related the Windows version:

Installation of Erlang in Windows

We can find the related Windows installer for RabbitMQ with the help of RabbitMQ download webpage as shown in the screenshot showing the download webpage of RabbitMQ. Then, we just need to click and install the RabbitMQ on our Windows computer. Besides installing using the installer, we can install using the Windows binary file that is served within the RabbitMQ download webpage. The following instructions will be enough for installing RabbitMQ without the installer:

  • Download the binary file for RabbitMQ Windows binary files

  • Extract the downloaded RabbitMQ zip file to our local folder

It is possible to install the RabbitMQ on your Windows computer in both ways. Note that you may add the RabbitMQ binaries directory to the windows system path in the system/environment variable settings.

Mac OS X

As we specified, we only have one requirement to install RabbitMQ on our computers. In Mac OS X, we have package managers and we have the opportunity to compile from the source for both Erlang runtime environment and RabbitMQ.

Firstly, both Erlang and RabbitMQ can be easily installed on Mac OS X using package managers. Although we have lots of package managers on Mac OS X, Homebrew and MacPort are the ones that are mostly used in Mac OS X. So, we'll talk about the installation using Homebrew and MacPorts.

Homebrew has both RabbitMQ and Erlang on its repository. As RabbitMQ has a dependency with Erlang, Homebrew finds its dependent software and installs them together.

Note

Homebrew is just another package manager for Mac OS X. Homebrew is quite easy to install on Mac OS X and has lots of packages. So, you would find your application in its repository. Check it out at http://brew.sh

So, we just need to install RabbitMQ in Homebrew using the brew install rabbitmq command on our terminal as shown in the following image:

Homebrew Installation of RabbitMQ

MacPorts has the similar method of operation with Homebrew. MacPorts also installs the software with its dependencies. Therefore, we just need to install RabbitMQ in MacPorts using the port install rabbitmq-server command on our terminal, as shown in the following image:

MacPorts Installation of RabbitMQ

Another way to install RabbitMQ and Erlang is by compiling the source codes in Mac OS X. Before compiling, we need the following to compile Erlang source code:

  • The GNU make

  • The GNU C compiler

  • Perl 5

After downloading and unzipping the source codes of the Erlang, we just need common commands on the Erlang folder for compiling from source code, as follows:

./configure
make
make install

Finally, we just need to download and unzip the RabbitMQ binary files.

Ubuntu

Ubuntu is just another Linux distribution based on Debian. Similar instructions as the ones we discussed for the installation on the Mac OS X would be applied for Ubuntu.

Ubuntu has a package manager called Advanced Packaging Tool (apt-get) and has a Debian package manager called dpkg. So, we are able to install RabbitMQ and Erlang runtime environment using apt-get. Moreover, similar to Mac OS X, we can compile from source codes of Erlang.

Firstly, as we said in the previous paragraph, we can install RabbitMQ using apt-get and dpkg. Before installing RabbitMQ, we should add the RabbitMQ repository to the APT repository using the following line (add the following line to /etc/apt/sources.list):

deb http://www.rabbitmq.com/debian/ testing main

Now, we are ready to install RabbitMQ and its dependency Erlang runtime environment, as shown in the following image:

sudo apt-get install rabbitmq-server

Ubuntu Installation of RabbitMQ

Also, we have the dpkg dependency management tool for installing RabbitMQ. RabbitMQ has packages for dpkg in its download webpage. We can download it from its website then run the following command:

dpkg –i rabbitmq-server.deb 

Secondly, we have another option, which was explained in the Mac OS X topic. That is, compiling from source codes. We just need to compile the downloaded Erlang source code, and we are ready to run the downloaded binary files of RabbitMQ. You can look at the details in the Mac OS X section.

Fedora

Fedora is yet another Linux distribution based on Red Hat. Installation instructions of Fedora are similar to Ubuntu's installation. Fedora has package managers called rpm and yum.

Firstly, we are able to install RabbitMQ with its dependency Erlang using package managers. Before using yum, we should run the following command to add RabbitMQ repository:

wget -O /etc/yum.repos.d/epel-erlang.repo http://repos.fedorapeople.org/repos/peter/erlang/epel-erlang.repo

Then, we can install RabbitMQ using the following command as shown in the following screenshot:

sudo yum install rabbitmq-server

Fedora Installation of RabbitMQ

Fedora has another package manager, which comes from Red Hat, called rpm. As RabbitMQ publishes package as rpm, we can easily install using the rpm package manager. After downloading the rpm package from RabbitMQ webpage, we can install RabbitMQ with Erlang, using following command:

rpm –ivh rabbitmq-server.rpm

Secondly, we have another option, which was explained in the Mac OS X topic. That is compiling from source code. We just need to compile the downloaded Erlang source code and we are ready to run the downloaded binary files of RabbitMQ. You can look at the details in the Mac OS X section.

Amazon elastic compute cloud (EC2)

Amazon Web Services (AWS) is cloud-computing platform offered from Amazon. AWS has lots of features for developers such as autoscaling. Besides RabbitMQ, AWS also offers their own messaging service called Simple Queue Service: however, RabbitMQ has lots of advantages over Amazon SQS. For instance, RabbitMQ has an extendable plugin system, whereas SQS capabilities are short. RabbitMQ implements a standard approach to message acknowledgement and consumption, whereas SQS has its own standards.

Anyway, we can easily install our RabbitMQ to the AWS EC2 instance and can save images of the RabbitMQ installed operating system. In AWS EC2, we choose one of the operating systems from the list or any other instance that we used earlier.

Tip

Amazon EC2 would be a good choice for your servers. In a scalable architecture, we need clusters of Message Brokers, databases, caches, and so on. EC2 gives you great API, and it's really quite easy to create clusters using EC2. We will talk about the clusters of RabbitMQ using EC2 in Chapter 4, Clustering and High Availability.

As we explained, we are able to install RabbitMQ on Windows, Linux, and Mac OS X, we just need to follow the instructions for AWS EC2 that are explained well in the preceding sections. Let's take a look at the following screenshot:

Amazon EC2

 

Starting RabbitMQ


As we can see, the installation part of RabbitMQ is quite easy and starting RabbitMQ is similar to its installation. Some package managers in Linux, Mac OS X, and Windows installer add configuration parameters to operation system's configuration for automatic startup. In such a case, we don't need to run the RabbitMQ command manually; however, if we install RabbitMQ manually, we need to run the RabbitMQ commands manually.

Starting RabbitMQ on Windows

If we use the Windows installer of RabbitMQ, the installer already makes configurations for starting automatically. Therefore, we don't need to run RabbitMQ manually; however, whenever we'd like to control the status of the server, we just need to run following command on the sbin folder of RabbitMQ:

rabbitmqctl status

Status of RabbitMQ in Windows

You may have installed RabbitMQ manually on your Windows and you might wonder how you can run the RabbitMQ server. You should run the following command to start RabbitMQ (you have to run this command with an administrative user). Moreover, you can install the RabbitMQ server as a Windows service:

rabbitmq-server

Other OSes (Linux, Mac OS X)

There's isn't much difference in running RabbitMQ on Windows and other operating systems. If we have RabbitMQ already installed using package managers, such as apt-get, yum, and so on, we don't need to run the RabbitMQ manually because RabbitMQ has already started automatically. So, we'd like to check the status of the RabbitMQ using the following command:

sudo rabbitmqctl status

Status of RabbitMQ in Unix

After controlling the RabbitMQ status, if we get a message that says that RabbitMQ isn't running, then we should run RabbitMQ using the following command on the sbin folder of RabbitMQ installation folder:

rabbitmq-server

Starting of RabbitMQ in Unix

 

Summary


We finally finished our first chapter, which introduces messaging concepts along with brief details about Message Queues and Message Brokers, and the protocol called AMQP that defines the functionalities of a Message Queue. Finally, our chapter introduces the RabbitMQ, providing information about its installation on different types of operating systems and how we can run the RabbitMQ server. Now, we are ready to jump into the details of RabbitMQ, starting from its configuration in the next chapter.

About the Authors

  • Emrah Ayanoglu

    Emrah Ayanoglu has been into technology since a young age, when he was programming with his Tandy 1000 using Basic language. His deep interest and passion for programming lead to him pursue computer engineering at Bilkent University, Ankara. He now works as a software engineer and heavily works on integrating software systems using RabbitMQ.

    He frequently speaks at different tech conferences about scalability and real-time web applications where RabbitMQ has a major role. Additionally, he participates in different open source projects.

    For the future, he hopes to participate more in open source projects and work on the real-time scalable applications.

    Browse publications by this author
  • Yusuf Aytaş

    Yusuf Aytaş is a software engineer. He completed his B.S. and M.S. in computer science from Bilkent University, Ankara. He has worked in both early stage startups and multinational companies. He is proficient in agile methodologies, continuous delivery, and software development best practices.

    Browse publications by this author
  • Dotan Nahum

     

    Browse publications by this author

Latest Reviews

(4 reviews total)
I normally buy books to avoid manuals. In this case, the books coverage o rabbit MQ administration is so shallow (ex: I had to read the manual to find out what the .* meant) that it's better to read the manuals. The only advantage to buying this book is that at least it gives you an idea (sequence) as to how to approach the manuals.
Good
Excellent

Recommended For You

RabbitMQ Cookbook

Knowing a reliable enterprise messaging system based on the AMQP standard can be an essential for today’s software developers. This cookbook helps you learn all the basics of RabbitMQ through recipes, code, and real-life examples.

By Sigismondo Boschi and 1 more
Mastering Object-Oriented Python - Second Edition

Gain comprehensive insights into programming practices, and code portability and reuse to build flexible and maintainable apps using object-oriented principles

By Steven F. Lott