What do you get with a Packt Subscription?

Free for first 7 days. $15.99 p/m after that. Cancel any time!

Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!

50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

Thousands of reference materials covering every tech concept you need to stay up to date.

Subscribe now

View plans & pricing

Apache Kafka 1.0 Cookbook

Chapter 1. Configuring Kafka

In this chapter, we will cover the following topics:

Installing Kafka
Running Kafka
Configuring Kafka brokers
Configuring Kafka topics
Creating a message console producer
Creating a message console consumer
Configuring the broker settings
Configuring threads and performance
Configuring the log settings
Configuring the replica settings
Configuring the Zookeeper settings
Configuring other miscellaneous parameters

Introduction

This chapter explains the basic recipes to get started with Apache Kafka. It discusses how to install, configure, and run Kafka. It also discusses how to make basic operations with a Kafka broker.

Kafka can run in several operating systems: Mac, Linux, and even Windows. As it usually runs in production on Linux servers, the recipes in this book are designed to run in Linux environments. This book also considers the bash environment usage.

Kafka scales very well in a horizontal fashion without compromising speed and efficiency.

This chapter explains how to install, configure, and run Kafka. As this is a practical recipes book, it does not cover the theoretical details of Kafka. These three things are enough theory for the moment:

To connect heterogeneous applications, we need to implement a mechanism for message publishing by sending and receiving messages among them. A message router is known as a message broker. Kafka is a software solution to deal with routing messages among consumers in a quick way.
The message broker has two directives: the first is to not block the producers, and the second is to isolate producers and consumers (the producers should not know who their consumers are).
Kafka is two things: a real-time, publish-subscribe solution, and a messaging system. Moreover, it is a solution: open source, distributed, partitioned, replicated, commit-log based, with a publish-subscribe schema.

Before we begin it is relevant to mention some concepts and nomenclature in Kafka:

Broker: A server process
Cluster: A set of brokers
Topic: A queue (that has log partitions)
Offset: A message identifier
Partition: An ordered and immutable sequence of records continually appended to a structured commit log
Producer: Those who publish data to topics
Consumer: Those who process the feed
ZooKeeper: The coordinator
Retention period: The time to keep messages available for consumption

In Kafka, there are three types of clusters:

Single node: Single broker
Single node: Multiple Broker
Multiple node: Multiple Broker

There are three ways to deliver messages:

Never redelivered: The messages may be lost
May be redelivered: The messages are never lost
Delivered once: The message is delivered exactly once

There are two types of log compaction:

Coarse grained: By time
Finer grained: By message

The next six recipes contain the necessary steps to make a full Kafka test from zero.

Installing Kafka

This is the first step. This recipe shows how to install Apache Kafka.

Getting ready

Ensure that you have at least 4 GB of RAM in your machine; the installation directory will be /usr/local/kafka/ for Mac users and /opt/kafka/ for Linux users. Create these directories.

How to do it...

Go to the Apache Kafka home page at http://kafka.apache.org/downloads, as in Figure 1-1, Apache Kafka download page:

Figure 1-1. Apache Kafka download page

The current available version of Apache Kafka is 0.10.2.1, as a stable release. A major limitation with Kafka since 0.8.x is that it is not backward-compatible. So, we cannot replace this version for one prior to 0.8. Once you've downloaded the latest available release, let's proceed with the installation.

Remember, for Mac users, replace the directory /opt/ for /usr/local/ in the examples.

Installing Java in Linux

We need Java 1.7 or later. Download and install the latest JDK from Oracle's website: http://www.oracle.com/technetwork/java/javase/downloads/index.html

Change the file mode:

> chmod +x jdk-8u131-linux-x64.rpm

Go to the directory in which you want to perform the installation:

> cd <directory path name>

Run the rpm installer with the command:

> rpm -ivh jdk-8u131-linux-x64.rpm

Finally, add the environment variable JAVA_HOME. This command will write the JAVA_HOME environment variable to the file /etc/profile:

> echo "export JAVA_HOME=/usr/java/jdk1.8.0_131" >> /etc/profile

Installing Scala in Linux

The following are the steps to install Scala in Linux:

Download the latest Scala binary from: http://www.scala-lang.org/download
Extract the downloaded file scala-2.12.2.tgz:

> tar xzf scala-2.12.2.tgz

Most tutorials agree that the best place to set environment variables is in the /etc/profile file.
Create the SCALA_HOME environment variable:

> export SCALA_HOME=/opt/scala

Add the Scala bin directory to the PATH variable:

> export PATH=$PATH:$SCALA_HOME/bin

Installing Kafka in Linux

The following are the steps to install Kafka in Linux:

Extract the downloaded file kafka_2.10-0.10.2.1.tgz:

> tar xzf kafka_2.10-0.10.2.1.tgz

Create the KAFKA_HOME environment variable:

> export KAFKA_HOME=/opt/kafka_2.10-0.10.2.1

Add the Kafka bin directory to the PATH variable:

> export PATH=$PATH:$KAFKA_HOME/bin

Now Java, Scala, and Kafka are installed.

There's more...

To do all these steps in command-line mode, there is a powerful tool for Mac users called brew (the equivalent in Linux would be yum).

To install from the command line, we use the following steps:

With brew, install sbt (Scala build tool):

> brew install sbt

If you already have it (downloaded in the past), upgrade it:

> brew upgrade sbt

The output is similar to:

> brew upgrade sbt
==> Upgrading 1 outdated package, with result:
sbt 0.13.15
==> Upgrading sbt
==> Using the sandbox
==> Downloading https://github.com/sbt/sbt/releases/download/v0.13.15/sbt-0.13.15.tgz
==> Downloading from https://github-cloud.s3.amazonaws.com/releases/279553/09838df4-23c6-11e7-9276-14
######################################################################## 100.0%
==> Caveats
You can use $SBT_OPTS to pass additional JVM options to SBT:
   SBT_OPTS="-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
This formula is now using the standard lightbend sbt launcher script.
Project specific options should be placed in .sbtopts in the root of your project.
Global settings should be placed in /usr/local/etc/sbtopts
==> Summary
/usr/local/Cellar/sbt/0.13.15: 378 files, 63.3MB, built in 1 minute 5 seconds

With brew, install Scala:

> brew install scala

If you already have it (downloaded in the past), upgrade it:

> brew upgrade scala

The output is similar to:

> brew install scala
==> Using the sandbox
==> Downloading https://downloads.lightbend.com/scala/2.12.2/scala-2.12.2.tgz
######################################################################## 100.0%
==> Downloading https://raw.githubusercontent.com/scala/scala-tool-support/0a217bc/bash-completion/sr
######################################################################## 100.0%
==> Caveats
To use with IntelliJ, set the Scala home to:
/usr/local/opt/scala/idea
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
/usr/local/Cellar/scala/2.12.2: 44 files, 19.9MB, built in 19 seconds
Mist:Downloads admin1$ scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

With brew, install Kafka (it also installs ZooKeeper):

> brew install kafka

If you already have it (downloaded in the past), upgrade it:

> brew upgrade kafka

The output is similar to:

> brew install kafka
==> Installing dependencies for kafka: zookeeper
==> Installing kafka dependency: zookeeper
==> Downloading https://homebrew.bintray.com/bottles/zookeeper-3.4.9.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring zookeeper-3.4.9.sierra.bottle.tar.gz
==> Using the sandbox
==> Caveats
To have launched start zookeeper now and restart at login:
brew services start zookeeper
Or, if you don't want/need a background service you can just run:
zkServer start
==> Summary
/usr/local/Cellar/zookeeper/3.4.9:
242 files, 18.2MB
==> Installing kafka
==> Downloading https://homebrew.bintray.com/bottles/kafka-0.10.2.0.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring kafka-0.10.2.0.sierra.bottle.tar.gz
==> Caveats
To have launchd start kafka now and restart at login:
brew services start kafka
Or, if you don't want/need a background service you can just run:
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties
==> Summary
/usr/local/Cellar/kafka/0.10.2.0: 145 files, 37.3MB

Configuring Kafka brokers

This recipe shows how to deal with the Kafka brokers' basic configuration. For learning and development purposes, one can run Kafka in standalone mode. The real Kafka power is unlocked when it is running with replication in cluster mode and the topics are partitioned accordingly.

There are two main advantages of the cluster mode: parallelism and redundancy. Parallelism is the capacity to run tasks simultaneously among the cluster members. Redundancy warrants that when a Kafka node goes down, the cluster is safe and accessible from the other nodes.

Single node clusters are not recommended for production environments, so this recipe shows how to configure a cluster with several nodes.

Getting ready

Go to the Kafka installation directory (/usr/local/kafka/ for Mac users and /opt/kafka/ for Linux users):

> cd /usr/local/kafka

How to do it...

As already said, a broker is a server's instance. This recipe shows how to start two different servers on one machine. There is a server configuration template called server.properties located in the Kafka installation directory in the config sub-directory:

For each Kafka broker (server) that we want to run, we make a copy of the configuration file template and rename it accordingly. In this example, the cluster is called synergy:

> cp config/server.properties synergy-1.properties
> cp config/server.properties synergy-2.properties

Modify each file according to the plan. If the file is called synergy-1, the broker.id should be 1. Specify the port in which the server should run; the recommendation is 9093 for synergy-1 and 9094 for synergy-2. The port property is not set in the template, so add the line accordingly. Finally, specify the location of the Kafka logs (specific archives to store all the Kafka broker operations); in this case, we use the directory /tmp.

In synergy-1.properties, set:

broker.id=1
port=9093
log.dir=/tmp/synergy-1-logs

In synergy-2.properties, set:

broker.id=2
port=9094
log.dir=/tmp/synergy-2-logs

Start the Kafka brokers using the kafka-server-start.sh command with the corresponding configuration file. Don't forget that Zookeeper must be already running with its corresponding Kafka node and the ports should not be in use by another process:

> bin/kafka-server-start.sh synergy-1.properties &
...
> bin/kafka-server-start.sh synergy-2.properties &

Recall that trailing & is to specify that you want your command line back. If you want to see the broker output, it is recommended that you run each command in its own command-line window.

How it works...

The properties file contains the server configuration. The server.properties file located in the config directory is just a template.

All of the members of the cluster should point to the same Zookeeper cluster. Every broker is identified inside the cluster by the name specified in the broker.id property. If the port property is not specified, Zookeeper will assign the same port number and will overwrite the data. If log.dir is not specified, all the brokers will write to the same default log.dir. If the brokers are going to run on different machines, then port and log.dir might not be specified.

There's more...

Before assigning a port to a server, there is a useful command to see what process is running on a specific port (in this case, the port 9093):

> lsof -i :9093

The output of the previous command is something like this:

COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
java   17479 admin   97u IPv6 0xcfbcde96aa59c3bf     0t0 TCP *:9093 (LISTEN)

Try to run this command before starting the Kafka servers and run it after starting to see the change. Also, try to start a broker on a port in use to see how it fails.

To run Kafka nodes on different machines, change the ZooKeeper connection string in the configuration file; its default value is:

zookeeper.connect=localhost:2181

This value is correct only if you are running the Kafka broker on the same machine as Zookeeper. In production, it could not happen. To specify that ZooKeeper is running on different machines (that is, in a ZooKeeper cluster), set:

zookeeper.connect=localhost:2181, 192.168.0.2:2183, 192.168.0.3:2182

The previous line says that Zookeeper is running on the localhost machine on port 2181, on the machine with IP Address 192.168.0.2 on port 2183, and on the machine with IP Address 192.168.0.3 on port 2182. The Zookeeper default port is 2181, so try to run it there.

As an exercise, try to raise a broker with incorrect information about Zookeeper. Also, in combination with the lsof command, try to raise Zookeeper on a port in use.