Kafka was created at LinkedIn. To start with LinkedIn used Java Message Service (JMS). But when they needed more power, that is, a scalable architecture, the LinkedIn development team decided to build the project that today we know as Kafka. In 2011, Kafka was open sourced as theApache Project. Due to size constraints, in this section we'll leave the reader with some reflections on why the architecture is designed in the way it is.
The Figure 5-6, A topic with 3 partitions, shows a topic with three partitions. We can see the five Kafka components: Zookeeper, broker, topic, producer and consumer.
Figure 5-6. A topic with 3 partitions
The Kafka project goals are:
- An API: To support the custom implementation of producers and consumers
- Low overhead: Low network latency and low storage overhead with message persistence on disk
- High throughput: Publishing and subscribing of millions of messages, supporting data feeds in real time
- Distributed: Highly scalable architecture to handle low...
