This is a turorial about the Apache Kafka Basic Operations.
1. Introduction
Apache Kafka is a streaming process software platform. In this example, I will show the following basic operations with a Windows computer:
- How to start/stop server
- How to create and list topics
- How to consume messages from a topic
2. Technologies Used
The example commands in this article uses:
- Apache Kafka 2.6
3. Manage Kafka Server
If you have not installed Apache Kafka yet, please click here to install.
3.1 Start Zookeeper
In this step, I will start zookeeper with the following command:
zookeeper-server-start
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>zookeeper-server-start.bat ..\..\config\zookeeper.properties
You should find the following log statement to confirm that the Zookeeper is started.
[2020-10-29 21:47:35,387] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
3.2 Start Kafka Broker
In this step, I will start a Kafka server with the following command:
kafka-server-start
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-server-start.bat ..\..\config\server.properties
You should find the following log statement to confirm that the Kafka broker is started.
[2020-10-29 21:48:35,692] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
4. Manage Kafka Topics
In this step, I will use kafka-topics command to list all the topics on the giving Kafka broker and create a new topic.
kafka-topic
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic zheng_test_topic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic zheng_test_topic.
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-topics.bat --list --zookeeper localhost:2181
zheng_test_topic
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>
5. Consume Messages from a Topic
In this step, I will use kafka-console-consumer command to receive messages from a giving topic at a giving broker.
kafka-console-consumer
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning
{"data", "some_value"}
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic zheng-cli-topic3 --from-beginning --property print.key=true
Key3 {"data", "some_value"}
Key2 {"data", "some_value"}
While listening on the topic, I will use the following command to send message to the topic:
kafka-console-producer
C:\MaryZheng\kafka_2.12-2.6.0\bin\windows>kafka-console-producer.bat --topic zheng_test_topic --bootstrap-server localhost:9092
When the prompt symbol “>” appears, you can type a message and hit the enter key to send. You should see the same message showing up on the consumer window too. See the screenshot for an example.
6. View Server Log
When troubleshooting issue, the first place to check is the server log. For mine, the log file is located at C:\MaryZheng\kafka_2.12-2.6.0\logs\
Here is the example error I got when testing different Kafka servers with different versions but all using the same data directory.
[2020-10-29 21:54:46,097] ERROR Shutdown broker because all log dirs in C:\tmp\kafka-logs have failed (kafka.log.LogManager)
This error can be fixed by cleaning the data directory and then restart the server.
7. Summary
In this example, I demonstrated the following Kafka operations which a Kafka application developer uses commonly:
- start/stop server
- list and create topics
- consume messages from a topic
