Building data pipelines with Kafka and NiFi

To build a data pipeline with Apache Kafka, you will need to create a producer since we do not have any production Kafka clusters to connect to. With the producer running, you can read the data like any other file or database.

The Kafka producer

The Kafka producer will take advantage of the production data pipeline from Chapter 11, Project — Building a Production Data Pipeline. The producer data pipeline will do little more than send the data to the Kafka topic. The following screenshot shows the completed producer data pipeline:

Figure 13.7 – The NiFi data pipeline

To create the data pipeline, perform the following steps:

1. Open a terminal. You need to create the topic before you can send messages to it in NiFi. Enter the following command:

   bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic users

   The preceding command is slightly different...