Networking with MQTT, Python, and the Mosquitto MQTT Broker

In the previous chapter, we created two Python servers and accompanying web pages using both a RESTful API and Web Socket approach to networking. In this chapter, we will cover another networking topology that is common in the IoT world, known as MQTT or Message Queue Telemetry Transport.

We will commence by setting up your development environment and installing the Mosquitto MQTT broker service on your Raspberry Pi. Then, we will learn about MQTT features using command-line tools that come with Mosquitto to help you to understand the core concepts in isolation. After that, we'll proceed to a Python IoT application that uses MQTT for its messaging layer—and yes, it'll be all about controlling the LED!

We will cover the following topics in this chapter:

• Installing the Mosquitto MQTT broker
• Learning...

Technical requirements

To perform the exercises in this chapter, you will need the following:

• Raspberry Pi 4 Model B
• Raspbian OS Buster (with desktop and recommended software)
• A minimum of Python version 3.5

These requirements are what the code examples in this book are based on. It's reasonable to expect that the code examples should work without modification on a Raspberry Pi 3 Model B or a different version of Raspbian OS as long as your Python version is 3.5 or higher.

You will find this chapter's source code in the chapter04 folder in the GitHub repository available at the following URL: https://github.com/PacktPublishing/Practical-Python-Programming-for-IoT

You will need to execute the following commands in a Terminal to set up a virtual environment and install the Python libraries required for the code in this chapter:

$ cd chapter04              # Change into this chapter's folder
$ python3 -m venv venv      # Create Python Virtual Environment
$ source...

Installing the Mosquitto MQTT broker

MQTT, or Message Queue Telemetry Transport, is a lightweight and simple messaging protocol targeted specifically for IoT applications. While a Raspberry Pi is powerful enough to leverage more complex messaging protocols, if you are using it as part of a distributed IoT solution, chances are you are going to encounter MQTT; hence, learning it is very important. Besides, its simplicity and open nature make it easy to learn and use.

Our introduction to MQTT is going to be performed using a popular open source MQTT broker called Mosquitto that we will install on your Raspberry Pi.

The examples we cover in this chapter were performed with the Mosquitto broker and client version 1.5.7, which are MQTT protocol version 3.1.1-complaint. A different version of the broker or client tools will be suitable as long as they are MQTT protocol version 3.1.x-compatible.

To install the Mosquitto MQTT broker service and client...

Learning MQTT by example

MQTT is a broker-based publishing and subscription messaging protocol (frequently paraphrased as pub/sub), while an MQTT broker (just like the Mosquitto MQTT broker we installed in the previous section) is a server that implements the MQTT protocol. By using an MQTT-based architecture, your applications can essentially hand off all complex messaging handling and routing logic to the broker so they can remain solution-focused.

MQTT clients (for example, your Python programs and the command-line tools we are about to use) create a subscription with the broker and subscribe to message topics they are interested in. Clients publish messages to a topic, and it is the broker that is then responsible for all message routing and delivery assurances. Any client may assume the role of a subscriber, a publisher, or both.

A simple conceptual MQTT-based system involving a pump, water tank, and controller application is illustrated in...

Publishing and subscribing MQTT messages

Let's work through the steps to send (that is, publish) and receive (that is, subscribe to) messages using MQTT:

1. In a Terminal, run the following command. mosquitto_sub (Mosquitto subscribe) is a command-line tool to subscribe to messages:

# Terminal #1 (Subscriber)
$ mosquitto_sub -v -h localhost -t 'pyiot'

The options are as follows:

• • -v (--verbose): verbose is so we get both the message topic and message payload printed on the Terminal.
  • -h (--host): localhost is the host of the broker we want to connect to; here it's the one we just installed. The default port used is 1883.
  • -t (--topic): pyiot is the topic we want to subscribe to and listen to.

In this chapter, we will require two and sometimes three Terminal sessions for the examples. The first line of a code block will indicate which Terminal you need to run a command in; for example, Terminal #1 in the preceding code block...

Exploring MQTT topics and wildcards

MQTT topics are used to categorize, or group, messages together in a hierarchical format. We have already been working with topics in our proceeding command-line examples, but in a non-hierarchical fashion. Wildcards, on the other hand, are special characters used by a subscriber to create flexible topic matching patterns.

Here are a few hierarchical topic examples from a hypothetical building with sensors. The hierarchy is delimited by the / character:

• level1/lounge/temperature/sensor1
• level1/lounge/temperature/sensor2
• level1/lounge/lighting/sensor1
• level2/bedroom1/temperature/sensor1
• level2/bedroom1/lighting/sensor1

There is no need to pre-create a topic on an MQTT broker. Using the default broker configuration (which we are), you just publish and subscribe to topics at will.

When the Mosquitto broker is configured to use authentication, there is the possibility to restrict access to topics based on a client ID and/or username...

Applying Quality of Service to messages

MQTT provides three Quality of Service (QoS) levels for individual message delivery—I am emphasizing individual message delivery because QoS levels apply to the delivery of individual messages and not to a topic. This will become clearer as you work through the examples.

While you, as the developer, stipulate the QoS for your messages, it's the broker that is responsible for ensuring that the message delivery adheres to the QoS. Here is the QoS you can apply to a message and what they mean for delivery:

QoS level	Meaning	Number of messages delivered
Level 0	The message will be delivered at most once, but maybe not at all.	0 or 1
Level 1	The message will be delivered at least once, but perhaps more.	1 or more
Level 2	The message will be delivered exactly once.	1

Table 2 – Message QoS levels

You might be asking the question: Level...

Retaining messages for later delivery

An MQTT broker can be instructed to retain messages published to a topic. Message retention comes in two flavors, known as retained messages and durable connections:

• A retained message is where the broker retains the last message published on a topic. This is also commonly referred to as the last known good message, and any client subscribing to a topic automatically gets this message. 
• Durable connections are also about retaining messages but in a different context. If a client tells the broker it wants a durable connection, then the broker retains QoS 1 and 2 messages for that client while it's offline.

Unless configured specifically, Mosquitto does not retain messages or connections across server restarts. To persist this information across a restart, a Mosquitto configuration file must contain the entry persistence true. A default installation of Mosquitto on a Raspberry Pi should include...

Publishing a retained message

A publisher can ask the broker to retain a message as the last known good message for a topic. Any newly connecting subscriber will immediately receive this last retained message.

Let's step through an example to demonstrate retained messages:

1. Run the following, noting that we're starting with Terminal #2, the publisher in this example:

# Terminal 2 (Publisher)
$ mosquitto_pub -r -q 2 -h localhost -t 'pyiot' -m 'hello, I have been retained!'

A new option has been added,-r (--retain), to tell the broker that this message should be retained for the topic.

Only a single retained message can exist for a topic. If you publish another message using the -r option, the previous retained message will be replaced.

2. Start a subscriber in another Terminal, and immediately you will receive the retained message:

# Terminal 1 (Subscriber)
$ mosquitto_sub -v -q 2 -h localhost -t 'pyiot'
pyiot hello, I have been...

Creating durable connections

A client subscribing to a topic can ask the broker to retain, or queue, messages for it while it's offline. This is known in MQTT terminology as a durable connection. For durable connections and delivery to work, the subscribing client needs to be configured and subscribe in a certain way, as follows:

• The client must provide a unique client ID to the broker when it connects.
• The client must subscribe with a QoS 1 or 2 (levels 1 and 2 guarantee delivery, but level 0 does not).
• The client is only guaranteed to get messages published with QoS 1 or 2.

The last two points concern an example where knowing QoS on both the publishing and subscribing sides of a topic is very important for IoT application design.

MQTT brokers can—and the default configuration of Mosquitto on the Raspberry Pi does—retain messages for durable connections between broker restarts.

Let's step through an example:

1. Start a subscriber, and then immediately...

Saying goodbye with a Will

Our final MQTT feature for exploration is known as a Will. A client (publisher or subscriber) can register a special Will message with the broker so that if the client dies and disconnects from the broker abruptly (for example, it loses its network connection or its batteries go flat), the broker on the clients' behalf will send out the Will message notifying subscribers of the device's demise.

Wills are just a message and topic combination similar to what we have been using previously. 

Let's see Wills in action, and for this, we're going to need three Terminals:

1. Open a Terminal and start a subscriber with the following command:

# Terminal #1 (Subscriber with Will)
$ mosquitto_sub -h localhost -t 'pyiot' --will-topic 'pyiot' --will-payload 'Good Bye' --will-qos 2 --will-retain

The new options are as follows:

• • --will-payload: This is the Will message.
  • --will-topic: This is the topic the Will...

Using MQTT broker services

There are several MQTT broker service providers on the internet that you can use to create MQTT-based messaging applications if you do not want to host your own MQTT broker. Many also offer free public MQTT brokers that you can use for testing and quick proofs-of-concept—but remember they are free and public, so do not publish any sensitive information!

If you experience frustration, disconnections, or unexpected behavior with a free public broker service, then test and verify your application with a local broker. You cannot reliably know or verify the traffic congestion, topic usage, or configuration details of an open public broker and how that may be impacting your application.

Here are a few free public brokers you can try. Just replace the -h localhost option in the preceding examples with the address of the broker. Visit the following pages for more information and instructions:

• https://test.mosquitto.org
• http://broker.mqtt-dashboard...

Introducing the Python Paho-MQTT client library

Before we get into Python code, we first need an MQTT client library for Python. At the start of this chapter in the Technical requirements section, we installed the Paho-MQTT client library, which was part of requirements.txt.

If you are new to MQTT and have not read the preceding section, Learning MQTT by example, I recommend stopping now and reading it first so you gain an understanding of MQTT concepts and terminology that will be used in the Python examples that follow. 

The Paho-MQTT client library comes from the Eclipse Foundation, which also maintains the Mosquitto MQTT broker. In the Further reading section, you will find a link to the official Paho-MQTT Client Library API documentation. After completing this chapter, if you wish to deepen your understanding of this library and its capabilities, I recommend reading through the official documentation and the examples found therein.

The Python Paho-MQTT library...

Controlling an LED with Python and MQTT

Previously, in the Installing the Mosquitto MQTT broker section, we tested the installation by visiting the http://localhost:8083 URL, which gave us a web page with a slider. However, at the time, we could not change the LED's brightness. When you moved the slider, the web page was publishing MQTT messages to the Mosquitto broker, but no program was receiving the messages to change the LED's brightness.

In this section, we'll see the Python code that subscribes to a topic called led and processes the messages generated by the slider. We will start by running the Python code and making sure we can change the LED's brightness.

Running the LED MQTT example

You will find the code in the chapter04/mqtt_led.py file. Please review this file before proceeding to get an overall idea of what it contains and then follow these steps:

1. Run the program in a Terminal with the following command:

# Terminal #1
(venv) $ python mqtt_led.py
INFO:main:Listening for messages on topic 'led'. Press Control + C to exit.
INFO:main:Connected to MQTT Broker

2. Now, open a second Terminal window and try the following, and the LED should turn on (be careful to make sure the JSON string is formed correctly):

# Terminal #2
$ mosquitto_pub -q 2 -h localhost -t 'led' -r -m '{"level": "100"}'

3. Did you notice the -r (--retain) option used in step 2? Terminate and restart mqtt_led.py and watch the log output in Terminal #1 and the LED. You should notice on startup that mqtt_led.py receives the LED's brightness value from the topic's retained...

Understanding the code

As we discuss the code found in chapter04/mqtt_led.py, pay particular attention to how the code connects to the MQTT broker and manages the connection life cycle. Furthermore, as we cover how the code receives and processes messages, try to relate the code workflow back to the command-line examples that we used to publish the message in the previous subsection, Running the LED MQTT example.

Once you have an understanding of our Python code and how it integrates with our MQTT broker, you'll have an end-to-end working reference solution built around MQTT messaging that you can adapt for your own needs and projects.

We will start by looking at the imports. As usual, we will skip over any common code that we have already covered in previous chapters, including logging setup and GPIOZero-related code.

Imports

The only new import we have in this example is for the Paho-MQTT client:

import paho.mqtt.client as mqtt  # (1)

At line (1), we are importing the Paho-MQTT client class and giving it the alias, mqtt. As mentioned previously, this is the client class that will allow us to create a full life cycle MQTT client in Python.

Next, we will consider global variables.

Global variables

The BROKER_HOST and BROKER_POST variables at line (2) are referring to our locally installed Mosquitto MQTT broker. Port 1883 is the standard default MQTT port:

# Global Variables
...
BROKER_HOST = "localhost"   # (2)
BROKER_PORT = 1883
CLIENT_ID = "LEDClient"     # (3)
TOPIC = "led"               # (4)
client = None # MQTT client instance. See init_mqtt()   # (5)
...

At line (3), we define CLIENT_ID, which will be the unique client identifier we use to identify our program with the Mosquitto MQTT broker. We must provide a unique ID to the broker so that we can use durable connections.

At line (4), we define the MQTT topic that our program will be subscribing to, while at line (5), the client variable is a placeholder that will be assigned the Paho-MQTT client instance, which we'll see shortly.

The set_led_level(data) method

set_led_level(data) at line (6) is where we integrate with GPIOZero to change the brightness of our LED and the method similar to the corresponding methods we covered in Chapter 3, Networking with RESTful APIs and Web Sockets Using Flask, so we will not cover the internals here again:

def set_led_level(data):  # (6)
   ...

The data parameter is expected to be a Python dictionary in the form of { "level": 50 }, where the integer is between 0 and 100 to indicate the brightness percentage.

Next, we have the callback functions for MQTT. We'll start by reviewing on_connect() and on_disconnect().

The on_connect() and on_disconnect() MQTT callback methods

The on_connect() and on_disconnect() callback handlers are examples of the full life cycle that is available using the Paho client class. We will see how to instantiate a Paho client instance and register these callbacks later when we cover the init_mqtt() method.

The parameters of interest to on_connect() at line (7) in the following code block are client, which is a reference to the Paho client class, and result_code, which is an integer describing the connection result. We see result_code used at line (8) to test the success of the connection. Notice the connack_string() method, which is used for a connection failure to translate result_code into a human-readable string.

When we speak of the MQTT client and see the client parameter at line (7) in the following code block, remember this is our Python code's client connection to the broker, NOT a reference to a client...

The on_message() MQTT callback method

It's the on_message() handler at line (11) that is called whenever a new message for a subscribed topic is received by our program. The message is available through the msg parameter, which is an instance of MQTTMessage. 

At line (12), we access the payload property of msg and decode it into a string. We expect our data to be a JSON string (for example, { "level": 100 }), so we parse the string into a Python dictionary using json.loads() and assign the result to data. If the message payload is not valid JSON, we catch the exception and log an error:

def on_message(client, userdata, msg):                    # (11)
    data = None
    try:                                                  
        data = json.loads(msg.payload.decode("UTF-8"))    # (12)
    except json.JSONDecodeError as e:
        logger.error("JSON Decode Error: " 
                   + msg.payload.decode("UTF-8")...

The init_mqtt() method

We see the Paho-MQTT client instance created and assigned to the global client variable at line (15). A reference to this object is the  client parameter we saw previously in the on_connect(), on_disconnect(), and on_message() methods.

The client_id parameter is set to be the client name we defined earlier in CLIENT_ID, while clean_session=False tells the broker that it must not clear any stored messages for our client when we connect. As we discussed earlier in the command-line examples, this is the back-to-front way of saying we want a durable connection so any messages published to the led topic are stored for our client when it's offline:

def init_mqtt():
    global client
    client = mqtt.Client(                                       # (15)
        client_id=CLIENT_ID,
        clean_session=False)

    # Route Paho logging to Python logging.
    client.enable_logger()                                      # (16)

    # Setup callbacks
    client.on_connect...

Main entry point

After initializing our LED and client instances, we get to the program's main entry point.

We are registering a signal handler to capture Ctrl + C key combinations at line (19). The signal_handler method (not shown) simply turns off our LED and gracefully disconnects from the broker:

# Initialise Module
init_led()
init_mqtt()

if __name__ == "__main__":
    signal.signal(signal.SIGINT, signal_handler)    # (19)
    logger.info("Listening for messages on topic '" 
       + TOPIC + "'. Press Control + C to exit.")

    client.loop_start()                             # (20)
    signal.pause()

At line (20), the call to client.loop_start() is what allows our client to start, connect to the broker, and receive messages.

Did you notice that the LED program is stateless? We are not storing or persisting any LED level in code or to disk. All our program does is subscribe to a topic on the broker and change the LED's brightness...

Building a web-based MQTT client

In Chapter 3, Networking with RESTful APIs and Web Sockets Using Flask, we covered a code example using Web Sockets, which included an HTML file and JavaScript web client. In this section, we will also be looking at a Web Socket-based web client built using HTML and JavaScript. However, this time, we will be leveraging the built-in Web Socket features provided by the Mosquitto MQTT broker and the compatible JavaScript Paho-JavaScript Web Sockets library (you will find a link to this library in the Further reading section).

For comparison, in Chapter 3, Networking with RESTful APIs and Web Sockets Using Flask, we created our Web Socket server ourselves in Python using Flask-SocketIO, while our web client used the Socket.io JavaScript Web socket library.

We interacted with the web client we are about to explore to control our LED previously in the  Installing the Mosquitto MQTT broker at section ...

Understanding the code

While the JavaScript library we are using in this example is different, you will find that the general structure and use of the JavsScript code are similar to the code we saw for the socket.io-based web client in Chapter 3, Networking with RESTful APIs and Web Sockets Using Flask. As usual, we will start by looking at the imports.

Imports

Our web client imports the Paho-MQTT JavaScript client library at line (1):

<title>MQTT Web Socket Example</title>
<script src="./jquery.min.js"></script>
<script src="./paho-mqtt.js"></script>  <!-- (1) -->

paho-mqtt.js can be also found in the chapter04/mosquitto_www folder.

The official documentation page for the Paho-MQTT JavaScript library is available at https://www.eclipse.org/paho/clients/js, while its official GitHub page is found at https://github.com/eclipse/paho.mqtt.javascript.

When you explore the Paho-MQTT JavaScript API further, start at its GitHub site and make note of any breaking changes that are mentioned. The documentation pages are known to contain code fragments that do not reflect the latest GitHub code base.

Next, we encounter the global variables.

Global variables

At line (2), we initialize a Client_ID constant that will identify our JavaScript client with the broker.

Each Paho JavaScript MQTT client must have a unique hostname, port, and client ID combination when it connects to the broker. To ensure we can run multiple web pages on a single computer for testing and demonstration, we use a random number to create a quasi-unique client ID for each web page:

<script type="text/javascript" charset="utf-8">
    messagePubCount = 0;
    const CLIENT_ID = String(Math.floor(Math.random() * 10e16)) // (2)
    const TOPIC   = "led";                                      // (3)

At line (3), we define the TOPIC constant with led, the name of the MQTT topic that we will be subscribing and publishing to shortly. Next, we create our client instance.

The Paho JavaScript MQTT client

At line (4), we create our Paho-MQTT Client instance and assign it to the client variable.

The parameters to Paho.MQTT.Client() are the broker's hostname and port. We are serving this web page via Mosquitto, so the broker's host and port will be the same as web pages:

const client = new Paho.Client(location.hostname,        // (4)
                               Number(location.port),
                               CLIENT_ID); 

You may have noticed in the http://localhost:8083 URL that the port is 8083, while in Python we used port 1883:

• Port 1883 is the MQTT protocol port on the broker. Our Python program connects directly to the broker on this port.
• We previously configured port 8083 as a Web Socket port on the Mosquitto broker. Web pages can speak HTTP and Web Socket protocols, not MQTT.

This raises an important point. While we're using the term MQTT in the context of our JavaScript code, we're really proxying the...

Connecting to the broker

We define our onConnectionSuccess() handler at line (5), which will be called after our client successfully connects to the broker. When we successfully connect, we then update the web page to reflect the successful connection and enable the slider control:

onConnectionSuccess = function(data) {         // (5)
    console.log("Connected to MQTT Broker");
    $("#connected").html("Yes");
    $("input[type=range].brightnessLevel")
          .attr("disabled", null);
                  
    client.subscribe(TOPIC);                   // (6)
};

client.connect({                               // (7)
   onSuccess: onConnectionSuccess,
   reconnect: true
 });       

Next, at line (6), we subscribe to the led topic. It's at line (7) that we connect to the broker. Notice that we're registering the onConnectionSuccess function as the onSuccess option.

Remember, similar to the Python example, always subscribe to...

The onConnectionLost and onMessageArrived handler methods

In the following code, at lines (8) and (9), we see how to register an onConnectionLost and onMessageArrived handler with our Paho-MQTT client instance:

client.onConnectionLost = function onConnectionLost(data) {    // (8)
  ...
}

client.onMessageArrived = function onMessageArrived(message) { // (9)
   ...
}

These two functions are similar in principle to their corresponding functions in the socket.io example from the previous Chapter 3, Networking with RESTful APIs and Web Sockets Using Flask, in that they update the slider and web page text based on the data found in their respective data and message parameters.

Next, we have our document ready function.

JQuery document ready function

Finally, we encounter the document ready function at line (1o) where we initialize our web page content and register the event listener for the slider:

$(document).ready(function() {                                   // (10)
    $("#clientId").html(CLIENT_ID);

    // Event listener for Slider value changes.
    $("input[type=range].brightnessLevel").on('input', function() {
        level = $(this).val();

        payload = {
            "level": level
         };

        // Publish LED brightness.
        var message = new Paho.Message(                         // (11)
           JSON.stringify(payload)
        );

        message.destinationName = TOPIC;                        // (12)
        message.qos = 2;
        message.retained = true;                                // (13)
        client.send(message);
    });
});

Within the sliders event handler at line (11), we see how to create an MQTT message. Notice the use of JSON.stringify...

Summary

In this chapter, we have explored and practiced the core concepts of MQTT. After installing and configuring the Mosquitto MQTT broker on your Raspberry Pi, we moved straight into learning a range of examples on the command line. We learned how to publish and subscribe to MQTT messages, how to understand topic construction and name hierarchies, and how we can attach a QoS level to a message.

We also covered durable connections and retained messages, two mechanisms offered by MQTT brokers for storing messages for later delivery. We concluded our walk-through of MQTT concepts by exploring a special message and topic type known as a Will, whereby a client can register a message with a broker that gets automatically published to a topic in cases where the client abruptly loses its connection.

Next, we reviewed and walked through a Python program that used the Paho Python MQTT library to subscribe to an MQTT topic and control the brightness of our LED in response to the messages...

Questions

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the book:

1. What is MQTT?
2. Your retained MQTT messages never get delivered. What should you check?
3. Under what condition will an MQTT broker publish a Will message?
4. You choose to use MQTT as your IoT application's messaging layer and must ensure that messages are sent and received. What is the minimum QoS level required?
5. You develop an application using MQTT and use the Mosquitto broker, but now you need to use a different broker. What does this mean for your code base and deployment configuration?
6. Where in code (hint: which handler method) should you subscribe to MQTT topics and why?

Further reading

We covered the basics of MQTT from an operational level in this chapter. If you want to learn more about MQTT from a protocol and data level, HiveMQ (an MQTT broker and service provider) has an excellent 11-part series on the MQTT protocol available at https://www.hivemq.com/blog/mqtt-essentials-part-1-introducing-mqtt.

The home page of the Mosquitto MQTT broker and client tools are available at the following URL:

• Mosquitto MQTT broker: https://mosquitto.org

The documentation and API references for the Paho-MQTT libraries we used in this chapter are available at the following URLs:

• Paho-MQTT Python library: https://www.eclipse.org/paho/clients/python
• Paho-MQTT JavaSctipt library: https://www.eclipse.org/paho/clients/js

In addition to MQTT, HTTP RESTful APIs, and Web Sockets, there are complimentary communication protocols that are specially designed for constrained devices, known as CoRA and MQTT-NS. The Eclipse Foundation has a summary of...