In this chapter, we will cover the following topics:
In the previous chapter, we explained how to program with the Apache Kafka publisher-subscriber messaging system. In Apache Kafka there are three types of clusters:
The following four recipes show how to run Apache Kafka in these clusters.
The first cluster configuration is single-node single-broker (SNSB). This cluster is very useful when a single point of entry is needed. Yes, its architecture resembles the singleton design pattern. A SNSB cluster usually satisfies three requirements:
If the proposed design has only one or two of these requirements, a redesign is almost always the correct option.
Sometimes, the single broker could become a bottleneck or a single point of failure. But it is useful when a single point of communication is needed.
Go to the Kafka installation directory (/usr/local/kafka/ for macOS users and /opt/kafka/ for Linux users):
> cd /usr/local/kafkaThe SNSB Kafka cluster is running; now let's create topics, producer, and consumer.
We need the previous recipe executed:
/usr/local/kafka/ for macOS users and /opt/kafka/ for Linux users):> cd /usr/local/kafkaThe following steps will show you how to create an SNSB topic, producer, and consumer.
SNSBTopic with one partition and one replica:> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic SNSBTopicWe obtain the following output:
Created topic "SNSBTopic".The command parameters are:
--replication-factor 1: This indicates just one replica--partition 1: This indicates just one partition--zookeeper localhost:2181: This indicates the ZooKeeper URLThe second cluster configuration is single-node multiple-broker (SNMB). This cluster is used when there is just one node but inner redundancy is needed.
When a topic is created in Kafka, the system determines how each replica of a partition is mapped to each broker. In general, Kafka tries to spread the replicas across all available brokers.
The messages are first sent to the first replica of a partition (to the current broker leader of that partition) before they are replicated to the remaining brokers.
The producers may choose from different strategies for sending messages (synchronous or asynchronous mode). Producers discover the available brokers in a cluster and the partitions on each (all this by registering watchers in ZooKeeper).
In practice, some of the high volume topics are configured with more than one partition per broker. Remember that having more partitions increases the I/O parallelism for writes and this increases the...
The SNMB Kafka cluster is running; now let's create topics, producer, and consumer.
We need the previous recipe executed:
/usr/local/kafka/ for macOS users and /opt/kafka/ for Linux users):> cd /usr/local/kafkaThe following steps will show you how to create an SNMB topic, producer, and consumer
SNMBTopic with two partitions and two replicas:> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic SNMBTopicThe following output is displayed:
Created topic "SNMBTopic".This command has the following effects:
Finally, the third cluster configuration is multiple-node multiple-broker (MNMB). This cluster is used when there are several nodes and one or many brokers per node.
Go to the Kafka installation directory (/usr/local/kafka/ for macOS users and /opt/kafka/ for Linux users):
> cd /usr/local/kafkaThe following diagram shows an example MNMB cluster:
Here we are presented with the real power of the cluster. In this cluster, Kafka should be installed on every machine in the cluster. Here, every physical server could have one or many brokers; all the nodes on the same cluster should connect to the same ZooKeeper.
The good news is that all the commands in the previous recipes remain the same. The commands for ZooKeeper, the broker, producer, and consumer, don't change.
© 2017 Packt Publishing Limited All Rights Reserved
