Download code from GitHub
Messaging or message queuing is a style of communication between applications or components that enables a loosely coupled architecture. Advanced Message Queuing Protocol (AMQP) is a specification that defines the semantics of an interoperable messaging protocol. RabbitMQ is an Erlang-based implementation of AMQP, which supports advanced features such as clustering.
In this chapter, we will cover the following topics:
Introducing concepts and terminology related to messaging
Discovering AMQP and RabbitMQ
Presenting the context in which all the book's examples will take place
Installing and configuring RabbitMQ
Smoke signals, couriers, carrier pigeons, and semaphores: if this was a riddle, you would think of messages right away. Humanity has always had the need to connect with each other, finding new ways to defy the challenge posed by the distance between the different groups of people who need to communicate. We've come a long way with modern technologies, but essentially the basics remain. Senders, recipients, and messages are the core of all our communication infrastructures.
Software applications have the same needs; systems need to exchange messages with each other. They sometimes need to be sure that the message sent has reached its destination. They sometimes need to receive an immediate response, but not all the time. In some cases, they may even need to receive more than one response. Based on these different needs, different styles of communication between systems have emerged.
All this can be explained with the help of the following figure:
The request-response style of interaction
This request-response style of interaction is the most common style; a system (acting as a client) interacts with a remote system (acting as a server) via a synchronous interaction with a remotely exposed communication endpoint. Whether it takes the form of a remote procedure call, a web service invocation, or consumption of a resource, the model is the same: one system sends a message to another and waits for the remote party to respond synchronously. Systems communicate with each other in a point-to-point manner.
The advantages of this approach are met with inconveniences. On one hand, programmers get a simple programming model as everything happens in a procedural fashion. On the other hand, the tight coupling between both parties has a deep impact on the architecture of the whole system as it is hard to evolve, hard to scale, and so on.
One-way interaction with message queuing
Enter the one-way style of interaction, where systems interact with each other in an asynchronous fashion via the transmission of messages, and generally through the intermediary of relaying parties known as message brokers. In this scheme, commonly referred to as messaging or message queuing, systems play the role of message publishers (producers) and message consumers. They publish a message to a broker on which they rely on to deliver it to the intended consumer. If a response is required, it will eventually come at some point on time through the same mechanism, but reversed (the consumer and producer roles will be swapped).
The advantage of the messaging approach is that systems are loosely coupled. They don't need to know exactly where they are located; a mere name is enough to reach them. Systems can, therefore, be evolved in an independent manner with no impact on each other as the reliability of message delivery is entrusted to a broker. This is demonstrated in the following figure:
Message enabling a loosely coupled architecture
Indeed, the architecture represented in the preceding figure allows the following:
The publishers or consumers fail without impacting each other
The performance of each side to leave the other side unaffected
The number of instances of publishers and consumers to grow and reduce and to accommodate their workload in complete independence
The publishers are unaware of the location and technology of the consumers and vice-versa
The main downside of this approach is that programmers cannot rely on the mental model of procedural programming where things immediately happen one after another. In messaging, things happen over time, so systems must be programmed to deal with it.
If all this is a little blurry, let's use an analogy of a well-known protocol: Simple Mail Transfer Protocol (SMTP). In this protocol, e-mails are published (sent) to an SMTP server. This initial server then stores and forwards the e-mail to the next SMTP server, and so on until the recipient e-mail server is reached. At this point, the message is queued in an inbox, waiting to be picked up by the consumer (typically, via POP3 or IMAP). With SMTP, the publisher has no idea when the e-mail will be delivered or whether it will eventually be delivered at all. In case of a delivery failure, the publisher can be notified of issues later down the line. The only sure fact is that the broker has successfully accepted the message it had initially sent.
Furthermore, if a response is needed, it will arrive asynchronously using the same delivery mechanism but with the publisher and consumer roles reversed. The entire process is demonstrated in the following figure:
The e-mail infrastructure as an analogy for message queuing
With these fundamental notions established, let's now delve into the messaging protocol that we are going to consider in this book: Advanced Message Queuing Protocol (AMQP).
The Advanced Message Queuing Protocol (AMQP) is an open standard that defines a protocol for systems to exchange messages. AMQP defines not only the interaction that happens between a consumer/producer and a broker, but also the over-the-wire representation of the messages and commands that are being exchanged. Since it specifies the wire format for messages, AMQP is truly interoperable—nothing is left to the interpretation of a particular vendor or hosting platform. And since it is open, the AMQP community has flourished with broker and client implementations in a wide range of languages.
Note
The AMQP 0-9-1 specification can be downloaded at http://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf.
Let's look at the following list of core concepts of AMQP, which we will revisit in detail in the upcoming chapters:
Broker: This is a middleware application that can receive messages produced by publishers and deliver them to consumers or to another broker.
Virtual host: This is a virtual division in a broker that allows the segregation of publishers, consumers, and all the AMQP constructs they depend upon, usually for security reasons (such as multitenancy).
Connection: This is a physical network (TCP) connection between a publisher/consumer and a broker. The connection only closes on client disconnection or in the case of a network or broker failure.
Channel: This is a logical connection between a publisher/consumer and a broker. Multiple channels can be established within a single connection. Channels allow the isolation of the interaction between a particular client and broker so that they don't interfere with each other. This happens without opening costly individual TCP connections. A channel can close when a protocol error occurs.
Exchange: This is the initial destination for all published messages and the entity in charge of applying routing rules for these messages to reach their destinations. Routing rules include the following: direct (point-to-point), topic (publish-subscribe) and fanout (multicast).
Queue: This is the final destination for messages ready to be consumed. A single message can be copied and can reach multiple queues if the exchange's routing rule says so.
Binding: This is a virtual connection between an exchange and a queue that enables messages to flow from the former to the latter. A routing key can be associated with a binding in relation to the exchange routing rule.
Overview of the concepts defined by the AMQP specification
You may have a message-queuing background and are by now wondering what are the main differences between AMQP and another protocol that you know. Here is a quick comparison of some of the main features:
Java Message Service (JMS): Unlike AMQP, this only defines the wire protocol for a Java programming interface and not messages. As such, JMS is not interoperable and only works when compatible clients and brokers are used. Moreover, unlike AMQP, it does not define the commands necessary to completely configure messaging routes, leaving too much room for vendor-specific approaches. Finally, in JMS, message producers target a particular destination (queue or topic), meaning the clients need to know about the target topology. In AMQP, the routing logic is encapsulated in exchanges, sparing the publishers from this knowledge.
MQ Telemetry Transport (MQTT): This is an extremely lightweight message-queuing protocol. MQTT focuses only on the publish-subscribe model. Like AMQP, it is interoperable and is very well suited for massive deployments in embedded systems. Like AMQP, it relies on a broker for subscription management and message routing. RabbitMQ can speak the MQTT protocol—thanks to an extension.
ØMQ (also known as ZeroMQ): This offers messaging semantics without the need for a centralized broker (but without the persistence and delivery guarantees that a broker provides). At its core, it is an interoperable networking library. Implemented in many languages, it's a tool of choice for the construction of high-performance and highly-available distributed systems.
Process inboxes: Programming languages and platforms such as Erlang or Akka offer messaging semantics too. They rely on a clustering technology to distribute messages between processes or actors. Since they are embedded in the hosting applications, they are not designed for interoperability.
Multiple commercial and open source implementations of AMQP are available. Often, existing messaging brokers have been extended with an AMQP adapter, like in the case of ActiveMQ.
The open source broker we will look at in detail for this book has been built from the ground to support AMQP. So let's now turn our focus on RabbitMQ.
RabbitMQ is an Erlang implementation of an AMQP broker. Erlang has been chosen to build it because of its intrinsic support for building highly-reliable and distributed applications. Indeed, it is used to run telecommunication switches for which a proverbial total system's availability of 9 nines has been reported (that's 32 milliseconds of downtime per year). Erlang is also able to run on any operating system.
RabbitMQ implements Version 0-9-1 of AMQP with custom extensions (as allowed by the protocol) and some undeprecations (for features RabbitMQ really wants to keep). For data persistence, it relies on Mnesia, the in-memory/file-persisted embedded database of Erlang, and specific message storage and index files. For clustering, it mainly relies on Erlang's ingrained clustering abilities. RabbitMQ can easily be extended with the addition of plugins; for example, a web-based administration console can be deployed on it, thanks to this mechanism. This is shown in the following diagram:
The RabbitMQ broker engaging in various topologies
As shown in the preceding figure, RabbitMQ brokers can not only be clustered together, they can also be connected together using different techniques, such as federation and shovels, in order to form messaging topologies with smart message routing across brokers and the capacity to span multiple data centers.
Note
What's the deal with AMQP 1.0?
AMQP 1.0 was published at the end of 2011 after the development and maintenance of AMQP was transferred to OASIS. Why hasn't RabbitMQ rushed to support this version, since it seems to be the first official release? The fact of the matter is that AMQP has been drastically revised between 0-9-1 and 1.0. It was so drastic that some core concepts, such as the exchange, no longer exist. So, AMQP 1.0 is a different protocol than 0-9-1, with no truly compelling reason to adopt it. It is not more capable than 0-9-1, and some would also argue that it has lost some of the key aspects that made it attractive in the first place.
We're done with our quick introduction to messaging, AMQP, and RabbitMQ. In the next section, we will introduce Clever Coney Media, a fictitious company that just discovered RabbitMQ and will put it to service for the greater good!
Clever Coney Media (CCM) is a fictitious integrated software and digital media agency that specializes in developing applications for online communities. Their software landscape, as shown in the following figure, is a hodgepodge of technologies:
Its flagship product is a Rich Internet Application (RIA), backed by a Java backend. It's used by end users to engage in thematic online communities.
The back office is built with Ruby on Rails.
The company's website and blog runs on PHP.
A bunch of ad hoc Python scripts are used to extract and message data in order to generate usage reports.
Clever Coney Media's heterogeneous software landscape
You may wonder, why is CCM looking at adding RabbitMQ to their already busy environment? The main driver for this is a new feature it wants to offer to their users, which is the capacity to send messages to other users. Think of it as a mix of chat, without the immediateness, and e-mail, without the long-running history. Instead of creating its own messaging infrastructure, it's decided to use a ready-made message oriented middleware like RabbitMQ.
We'll see in the rest of the book that as its knowledge and usage of RabbitMQ increases, CCM will discover new opportunities to leverage it in its environment. But for now, enough with the ado; let's follow CCM as it gets started with its very first step with RabbitMQ.
To get started, we will go through the following three installation and configuration steps:
Installing the RabbitMQ broker
Installing the management plugin
Configuring the vhost and user
CCM runs its production servers on Ubuntu Linux. Most of the developers' workstations run Mac OS X and Linux, while some run Windows. This heterogeneity is not a concern for RabbitMQ, which can run natively on all these operating systems.
RabbitMQ provides complete online installation guides for all the supported operating systems (you can access these at http://www.rabbitmq.com/download.html). In our case, we will follow the instructions for Debian/Ubuntu.
For greater control, we do not wish to use the RabbitMQ APT repository; instead, we want to download the Debian package and manually install it, as follows:
$ wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.1/rabbitmq-server_3.2.1-1_all.deb $ sudo dpkg -i rabbitmq-server_3.2.1-1_all.deb $ sudo apt-get -f --force-yes --yes install
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
In case you wonder why we first ran the dkpg software followed by the apt-get command, the reason is simple: the first attempt expectedly fails because none of the Erlang dependencies are present in our system. This failure generates a list of unresolved dependencies that the apt-get command picks up and installs, including the installation of the RabbitMQ broker.
Note
The installation of RabbitMQ has also installed Erlang on your machine. Though not absolutely required for using RabbitMQ, we want to encourage you to discover this simple yet powerful language and platform. You can learn more about Erlang at http://www.erlang.org/. You can also consider Elixir as an alternative language for the Erlang VM at http://elixir-lang.org.
We can verify that the RabbitMQ broker is actually working using the standard service command:
$ sudo service rabbitmq-server status
Status of node 'rabbit@ip-172-31-31-18' ...
[{pid,4027},
{running_applications,[{rabbit,"RabbitMQ","3.2.1"},
{mnesia,"MNESIA CXC 138 12","4.5"},
{os_mon,"CPO CXC 138 46","2.2.7"},
{xmerl,"XML parser","1.2.10"},
{sasl,"SASL CXC 138 11","2.1.10"},
{stdlib,"ERTS CXC 138 10","1.17.5"},
{kernel,"ERTS CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [rq:1] [async-threads:30] [kernel-poll:true]\n"},
{memory,[{total,27085496},
{connection_procs,2648},
{queue_procs,5296},
{plugins,0},
{other_proc,9040296},
{mnesia,57776},
{mgmt_db,0},
{msg_index,25768},
{other_ets,752416},
{binary,1952},
{code,14546600},
{atom,1360921},
{other_system,1291823}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,247537664},
{disk_free_limit,50000000},
{disk_free,7020503040},
{file_descriptors,[{total_limit,924},
{total_used,3},
{sockets_limit,829},
{sockets_used,1}]},
{processes,[{limit,1048576},{used,122}]},
{run_queue,0},
{uptime,73}]
...done.If you're wondering what format is used to represent the server status information, it's not JSON but in fact, Erlang lists and tuples. You can notice how the status data contains a lot of contextual information about RabbitMQ and the Erlang VM.
Tip
The default folders where the package has installed files are /etc/rabbitmq for configuration files, /usr/lib/rabbitmq for application files, and /var/lib/rabbitmq for data files.
If you take a look at the running processes for RabbitMQ, you'll find both the service wrapper and the Erlang virtual machine (also known as BEAM) running as follows:
$ pgrep -fl rabbitmq
3633 /bin/sh /usr/sbin/rabbitmq-server
3647 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@pegasus -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@pegasus.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@pegasus-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@pegasus-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@pegasus"Note
You may find that when RabbitMQ runs, a process named epmd is also running. This is the Erlang Port Mapper Daemon in charge of coordinating Erlang nodes in a cluster. It is expected to start even if you are not running clustered RabbitMQ.
Note that by default, the broker service is configured to auto-start when the Linux host starts. You can confirm and configure this via a tool called rcconf as shown in the following screenshot:
The RabbitMQ server service auto-starts by default
By default, RabbitMQ does not embed a web-based management console but offers it as an optional plugin. This management console makes it very easy to peek into a running RabbitMQ instance, so we definitely want to have it installed from the get go.
The Debian package has installed several scripts, one of them being rabbitmq-plugins, whose purpose is to allow the installation and removal of plugins. Let's use it to install the management plugin as follows:
$ sudo rabbitmq-plugins enable rabbitmq_management The following plugins have been enabled: mochiweb webmachine rabbitmq_web_dispatch amqp_client rabbitmq_management_agent rabbitmq_management Plugin configuration has changed. Restart RabbitMQ for changes to take effect.
Yes, it is that easy! As invited by the installer, we need to restart RabbitMQ as follows:
$ sudo service rabbitmq-server restart * Restarting message broker rabbitmq-server [ OK ]
Using our favorite web browser, we can now reach the home page of the management console by navigating to http://<hostname>:15672 as shown in the following screenshot:
The login screen of the management console
So, what Username and Password can we use to log in to the console? None yet, but we're going to to have a remedy for that too!
One of the scripts installed by the Debian package is rabbitmqctl, which is the RabbitMQ broker control script. It is of paramount importance as it is used to configure all aspects of the broker. We will now use it to configure an administration user in the broker as follows:
$ sudo rabbitmqctl add_user ccm-admin hare123 Creating user "ccm-admin" ... ...done. $ sudo rabbitmqctl set_user_tags ccm-admin administrator Setting tags for user "ccm-admin" to [administrator] ... ...done.
Tip
By default, RabbitMQ comes with a guest user authenticated with the guest password. You'll want to change this password to something else as follows:
sudo rabbitmqctl change_password guest guest123
By navigating back to the management console login screen, we are now able to log in with ccm-admin and hare123. We are welcomed by this overview of the broker's internals, as shown in the following screenshot:
The main dashboard of the management console
Note that at this point, the ccm-admin user is not able to introspect any exchange or queue in any virtual host. We will address this issue in a moment. But for now, we need another user for development purposes so that our applications can connect to RabbitMQ. So, let's create the ccm-dev user as follows:
$ sudo rabbitmqctl add_user ccm-dev coney123 Creating user "ccm-dev" ... ..done.
As discussed earlier in this chapter, RabbitMQ supports the notion of virtual hosts, which are logical subdivisions of its execution space. We're going to create a virtual host, also known as
vhost, for the development environment. So, anything that happens in it happens in isolation from any other environments we can create in the future (such as a QA environment). So, let's create a vhost named ccm-dev-vhost as follows:
$ sudo rabbitmqctl add_vhost ccm-dev-vhost Creating vhost "ccm-dev-vhost" ... ..done.
Tip
RabbitMQ comes with a default vhost named / on which the guest user has full permissions. Though this is convenient for quick tests, we recommend that you create dedicated vhosts in order to keep concerns separated so that it is possible to completely drop a vhost and restart from scratch without unexpected impacts.
As it currently is, neither the ccm-admin nor ccm-dev users have permission to do anything on ccm-dev-vhost. Let's fix this by giving the vhost full rights on it as follows:
$ sudo rabbitmqctl set_permissions -p ccm-dev-vhost ccm-admin ".*" ".*" ".*" Setting permissions for user "ccm-admin" in vhost "ccm-dev-vhost" ...done. $ sudo rabbitmqctl set_permissions -p ccm-dev-vhost ccm-dev ".*" ".*" ".*" Setting permissions for user "ccm-dev" in vhost "ccm-dev-vhost" ...done.
What have we just done? Most of the command is straightforward but the ".*" ".*" ".*" part looks a tad mysterious, so let's analyze it. It is a triplet of permissions for the considered vhost that respectively grant configure, write, and read permissions on the designated resources for the considered user and vhost. Resources, which consist of exchanges and queues, are designated by regular expressions that match their names. Thus, in our case, we are allowing any resource via the .* regular expression.
The actual commands that are granted depend on the resource type and the granted permissions. The reader can get a complete list of the access control policies supported by RabbitMQ at http://www.rabbitmq.com/access-control.html.
As an alternative to all command lines, you can also turn to the user management features of the management console. If you click on the Admin tab of the console and then on the ccm-dev user listed in the Users tab, you'll see what's shown in the following screenshot. The entire user configuration you've set from the command line is visible and editable in the management console.
Details of an individual user in the management console
A lot of ground was covered in this chapter. We learned about the architectural and design promises of messaging and how AMQP and RabbitMQ deliver on these promises. We discovered the reason why Clever Coney Media decided to introduce RabbitMQ in their software landscape. Finally, we installed and configured a RabbitMQ broker.
It's time to hit the ground running and write some code. You can now turn to the next chapter to start building your first RabbitMQ-powered application!
