Apache Kafka Command Line Interface (CLI)
Here are some commands often be used when we work with
Apache Kafka command line interface (CLI).
1. Start the Kafka server
We needs 2 steps:
1.1 Start the ZooKeeper
|
|
bin/zookeeper-server-start.sh config/zookeeper.properties
|
1.2. Start the Kafka server
|
|
bin/kafka-server-start.sh config/server.properties
|
2. List all topics
|
|
bin/kafka-topics.sh --zookeeper localhost:2181 --list
|
The output in my console:
|
|
hello-topic
my-replicated-topic
|
3. Create a topic
|
|
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-topic
|
The output is:
|
|
Created topic "hello-topic".
|
Above command will create a “
hello-topic“, with
replication-factor = 1 and the number of
partitions is
1. Note that the
replication-factor
controls how many servers will replicate each message that is written;
therefore, it should be less than or equal the number of Kafka
servers/brokers.
4. Describe a topic
|
|
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello-topic
|
The output is as below:
|
|
Topic:hello-topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
|
5. Publish messages to a topic
One of the most interesting command.
|
|
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
|
After this command, you can add any messages to the console, line by line. For ex:
You can stop the console consumer via
Ctrl-C
6. Consume messages
Below command will consume messages from the topic:
hello-topic
|
|
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
|
The output on my console is:
You can stop the console consumer via
Ctrl-C
7. Alter Apache Kafka Topics
7.1. Add more partitions to the topic
Below command will add 10 more partitions to the
hello-topic topic. Note that before, the topic has only 1 partition.
|
|
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 11 --topic hello-topic
|
The output in my console:
|
|
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
|
7.2. Delete a topic
|
|
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic hello-topic
|
7.3. Add configurations to the Kafka topic
The general syntax is:
|
|
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic kafkatopic --config <key>=<value>
|
For example, below command will set the
max message size = 128000 bytes for the
hello-topic topic.
|
|
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --config max.message.bytes=128000
|
Here is an example output:
|
|
WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.
Going forward, please use kafka-configs.sh for this functionality
Updated config for topic "hello-topic".
|
7.4. Add configurations to the Kafka topic
To remove above overridden configuration, we can use command:
|
|
bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --delete-config max.message.bytes
|
For the list of configurations, please reference
Apache Kafka page
