Enterprise Java

Apache Kafka Basic Operations Tutorial

This is a turorial about the Apache Kafka Basic Operations.

1. Introduction

Apache Kafka is a streaming process software platform. In this example, I will show the following basic operations with a Windows computer:

  • How to start/stop server
  • How to create and list topics
  • How to consume messages from a topic

2. Technologies Used

The example commands in this article uses:

  • Apache Kafka 2.6

3. Manage Kafka Server

If you have not installed Apache Kafka yet, please click here to install.

3.1 Start Zookeeper

In this step, I will start zookeeper with the following command:

zookeeper-server-start

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>zookeeper-server-start.bat ..\..\config\zookeeper.properties

You should find the following log statement to confirm that the Zookeeper is started.

[2020-10-29 21:47:35,387] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)

3.2 Start Kafka Broker

In this step, I will start a Kafka server with the following command:

kafka-server-start

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-server-start.bat ..\..\config\server.properties

You should find the following log statement to confirm that the Kafka broker is started.

[2020-10-29 21:48:35,692] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

4. Manage Kafka Topics

In this step, I will use kafka-topics command to list all the topics on the giving Kafka broker and create a new topic.

kafka-topic

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic zheng_test_topic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic zheng_test_topic.

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
zheng_test_topic

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>

5. Consume Messages from a Topic

In this step, I will use kafka-console-consumer command to receive messages from a giving topic at a giving broker.

kafka-console-consumer

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning
{"data", "some_value"}
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning --property print.key=true
Key3    {"data", "some_value"}
Key2    {"data", "some_value"}

While listening on the topic, I will use the following command to send message to the topic:

kafka-console-producer

C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-producer.bat --topic zheng_test_topic --bootstrap-server localhost:9092

When the prompt symbol “>” appears, you can type a message and hit the enter key to send. You should see the same message showing up on the consumer window too. See the screenshot for an example.

Kafka Operations - Consume Messages
Figure 1 Consume Messages

6. View Server Log

When troubleshooting issue, the first place to check is the server log. For mine, the log file is located at C:\MaryZheng\kafka_2.12-2.6.0\logs\

Here is the example error I got when testing different Kafka servers with different versions but all using the same data directory.

[2020-10-29 21:54:46,097] ERROR Shutdown broker because all log dirs in C:\tmp\kafka-logs have failed (kafka.log.LogManager)

This error can be fixed by cleaning the data directory and then restart the server.

7. Summary

In this example, I demonstrated the following Kafka operations which a Kafka application developer uses commonly:

  • start/stop server
  • list and create topics
  • consume messages from a topic

Mary Zheng

Mary has graduated from Mechanical Engineering department at ShangHai JiaoTong University. She also holds a Master degree in Computer Science from Webster University. During her studies she has been involved with a large number of projects ranging from programming and software engineering. She works as a senior Software Engineer in the telecommunications sector where she acts as a leader and works with others to design, implement, and monitor the software solution.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button