Kafka on Mesos

This section will introduce Kafka and explain how to set it up on Mesos while also discussing the problems commonly encountered during the setup process.

Introduction to Kafka

Kafka is a distributed publish-subscribe messaging system designed for speed, scalability, reliability, and durability. Some of the key terms used in Kafka are given as follows:

• Topics: These are the categories where message feeds are maintained by Kafka

• Producers: These are the upstream processes that send messages to a particular Kafka topic

• Consumers: These are the downstream processes that listen to the incoming messages in a topic and process them as per requirements

• Broker: Each node in a Kafka cluster is called a broker

Take a look at the following high-level diagram of Kafka (source: http://kafka.apache.org/documentation.html#introduction):

A partitioned log is maintained by the Kafka cluster for every topic, which looks similar to the following (source: http://kafka.apache.org/documentation.html...