Chapter 9: Monitoring Data Pipelines

You now know how to build data pipelines that are production ready—they are idempotent, use atomic transactions, and are version controlled. They are ready to start running in production, but before you can deploy them, you need to be able to monitor them. There will be errors in the code, in the data, in the network, or in other areas outside your control. You will need to know when errors happen, and when things are running as they should.

In this chapter, we're going to cover the following main topics:

• Monitoring NiFi in the GUI
• Monitoring NiFi using processors
• Monitoring NiFi with Python and the REST API

Monitoring NiFi using the GUI

The NiFi GUI provides several ways to monitor your data pipelines. Using the GUI is the simplest way to start monitoring your NiFi instance.

Monitoring NiFi with the status bar

Much of the information you need is on the status bar. The status bar is below the component toolbar and looks like the following screenshot:

Figure 9.1 – Component and status toolbars

Starting at the left of the status bar, let's look at what is being monitored:

• Active thread: This lets you know how many threads are running. You can get a sense of tasks and load.
• Total queued data: The number of flowfiles and the combined size on disk.
• Transmitting remote process groups and not transmitting remote process groups: You can run NiFi on multiple machines or instances on the same machine and allow process groups to communicate. These icons tell you whether they are or are not communicating.
• Running components, stopped components...

Monitoring NiFi with processors

Instead of relying on watching the NiFi GUI, you can insert a processor into your data pipeline to report what is happening with the pipeline. For example, you can use the PutSlack processor to send messages on failures or success.

To send Slack messages, you will need to create an app in your Slack workspace. You can do this by browsing to https://api.slack.com/apps. Click Create New App, as shown in the following screenshot:

Figure 9.14 – Creating a new app

Slack will ask you to name your app and then select a workspace, as shown in the following screenshot:

Figure 9.15 – Specifying a name and workspace for your app

When finished, you will be redirected to the app page. Under the Features heading, click Incoming Webhooks and turn it on, as shown in the following screenshot:

Figure 9.16 – Activating Incoming Webhooks

You will be asked to select a channel...

Using Python with the NiFi REST API

Using Python and the NiFi REST API, you could write your own monitoring tools, or wire up a dashboard. The NiFi REST API documentation is located at https://nifi.apache.org/docs/nifi-docs/rest-api/index.html. You can see all of the different endpoints by type and some information about each of them. This section will highlight some of the endpoints that you have covered in this chapter but by using the GUI.

The first thing we can look at are the system diagnostics. System diagnostics will show you your resource usage. You can see heap size, threads, repository usage, and several other metrics. To call the endpoint with requests, you can use the following code:

r=requests.get('http://localhost:9300/nifi-api/system-diagnostics')
data=r.json()
data['systemDiagnostics']['aggregateSnapshot']['maxHeap']
#'512 MB'
data['systemDiagnostics']['aggregateSnapshot']['totalThreads...

Summary

In this chapter, you have learned how to use the NiFi GUI to monitor your data pipelines using the status bar, the bulletin, and counters. You also learned how to add processors that can send information to you inside your data pipeline. With the PutSlack processor, you were able to send yourself direct messages when there was a failure, and you passed data from the flowfile in the message with the NiFi expression language. Lastly, you learned how to use the API to write your own monitoring tools and grab the same data as is in the NiFi GUI—even reading the contents of a single flowfile.

In the next chapter, you will learn how to deploy your production pipelines. You will learn how to use processor groups, templates, versions, and variables to allow you to import data pipelines to a production NiFi instance with minimal configuration.