Building a NiFi cluster

In this book, you have built a Kafka cluster, a ZooKeeper cluster, and a Spark cluster. Instead of increasing the power of a single server, through clustering, you are able to add more machines to increase the processing power of a data pipeline. In this chapter, you will learn how to cluster NiFi so that your data pipelines can run across multiple machines.

In this appendix, we're going to cover the following main topics:

• The basics of NiFi clustering
• Building a NiFi cluster
• Building a distributed data pipeline
• Managing the distributed data pipeline

The basics of NiFi clustering

Clustering in Apache NiFi follows a Zero-Master Clustering architecture. In this type of clustering, there is no pre-defined master. Every node can perform the same tasks, and the data is split between them. NiFi uses Zookeeper when deployed as a cluster.

Zookeeper will elect a Cluster Coordinator. The Cluster Coordinator is responsible for deciding whether new nodes can join – the nodes will connect to the coordinator – and to provide the updated flows to the new nodes.

While it sounds like the Cluster Coordinator is the master, it is not. You can make changes to the data pipelines on any node and they will be replicated to all the other nodes, meaning a non-Cluster Coordinator or a non-Primary Node can submit changes.

The Primary Node is also elected by Zookeeper. On the Primary Node, you can run isolated processes. An isolated process is a NiFi processor that runs only on the Primary Node. This is important because think what...

Building a NiFi cluster

In this section, you will build a two-node cluster on different machines. Just like with MiNiFi, however, there are some compatibility issues with the newest versions of NiFi and Zookeeper. To work around these issues and demonstrate the concepts, this chapter will use an older version of NiFi and the pre-bundled Zookeeper. To build the NiFi cluster, perform the following steps:

1. As root, or using sudo, open your /etc/hosts file. You will need to assign names to the machines that you will use in your cluster. It is best practice to use a hostname instead of IP addresses. Your hosts file should look like the following example:

   127.0.0.1	localhost
   ::1		localhost
   127.0.1.1	pop-os.localdomain	pop-os
   10.0.0.63	nifi-node-2
   10.0.0.148	nifi-node-1

2. In the preceding hosts file, I have added the last two lines. The nodes are nifi-node-1 and nifi-node-2 and you can see that they have different IP addresses. Make these changes in the hosts file for each machine...

Building a distributed data pipeline

Building a distributed data pipeline is almost exactly the same as building a data pipeline to run on a single machine. NiFi will handle the logistics of passing and recombining the data. A basic data pipeline is shown in the following screenshot:

Figure 16.4 – A basic data pipeline to generate data, extract attributes to json, and write to disk

The preceding data pipeline uses the GenerateFlowFile processor to create unique flowfiles. This is passed downstream to the AttributesToJSON processor, which extracts the attributes and writes to the flowfile content. Lastly, the file is written to disk at /home/paulcrickard/output.

Before running the data pipeline, you will need to make sure that you have the output directory for the PutFile processor on each node. Earlier, I said that data pipelines are no different when distributed, but there are some things you must keep in mind, one being that PutFile will write...

Managing the distributed data pipeline

The preceding data pipeline runs on each node. To compensate for that, you had to create the same path on both nodes for the PutFile processor to work. Earlier, you learned that there are several processors that can result in race conditions – trying to read the same file at the same time – which will cause problems. To resolve these issues, you can specify that a processor should only run on the Primary Node – as an isolated process.

In the configuration for the PutFile processor, select the Scheduling tab. In the dropdown for Scheduling Strategy, choose On primary node, as shown in the following screenshot:

Figure 16.6 – Running a processor on the Primary Node only

Now, when you run the data pipeline, the files will only be placed on the Primary Node. You can schedule processors such as GetFile or ExecuteSQL to do the same thing.

To see the load of the data pipeline on each node, you...

Summary

In this Appendix, you learned the basics of NiFi clustering, as well as how to build a cluster with the embedded Zookeeper and how to build distributed data pipelines. NiFi handles most of the distribution of data; you only need to keep in mind the gotchas – such as race conditions and the fact that processors need to be configured to run on any node. Using a NiFi cluster allows you to manage NiFi on several machines from a single instance. It also allows you to process large amounts of data and have some redundancy in case an instance crashes.