Apache Kafka Basic Operations Tutorial

This is a turorial about the Apache Kafka Basic Operations.

1. Introduction

Apache Kafka is a streaming process software platform. In this example, I will show the following basic operations with a Windows computer:

2. Technologies Used

The example commands in this article uses:

3. Manage Kafka Server

If you have not installed Apache Kafka yet, please click here to install.

3.1 Start Zookeeper

In this step, I will start zookeeper with the following command:

zookeeper-server-start

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>zookeeper-server-start.bat ..\..\config\zookeeper.properties

You should find the following log statement to confirm that the Zookeeper is started.

[2020-10-29 21:47:35,387] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)

3.2 Start Kafka Broker

In this step, I will start a Kafka server with the following command:

kafka-server-start

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-server-start.bat ..\..\config\server.properties

You should find the following log statement to confirm that the Kafka broker is started.

[2020-10-29 21:48:35,692] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

4. Manage Kafka Topics

In this step, I will use kafka-topics command to list all the topics on the giving Kafka broker and create a new topic.

kafka-topic

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic zheng_test_topic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic zheng_test_topic.
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
zheng_test_topic
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>

5. Consume Messages from a Topic

In this step, I will use kafka-console-consumer command to receive messages from a giving topic at a giving broker.

kafka-console-consumer

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning
{"data", "some_value"}
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning --property print.key=true
Key3    {"data", "some_value"}
Key2    {"data", "some_value"}

While listening on the topic, I will use the following command to send message to the topic:

kafka-console-producer

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-producer.bat --topic zheng_test_topic --bootstrap-server localhost:9092

When the prompt symbol “>” appears, you can type a message and hit the enter key to send. You should see the same message showing up on the consumer window too. See the screenshot for an example.

Figure 1 Consume Messages

6. View Server Log

When troubleshooting issue, the first place to check is the server log. For mine, the log file is located at C:\MaryZheng\kafka_2.12-2.6.0\logs\

Here is the example error I got when testing different Kafka servers with different versions but all using the same data directory.

[2020-10-29 21:54:46,097] ERROR Shutdown broker because all log dirs in C:\tmp\kafka-logs have failed (kafka.log.LogManager)

This error can be fixed by cleaning the data directory and then restart the server.

7. Summary

In this example, I demonstrated the following Kafka operations which a Kafka application developer uses commonly:

Exit mobile version