Kafka Commands

Apache Kafka Command Line Interface (CLI)

Here are some  commands often be used when we work with Apache Kafka command line interface (CLI).

1. Start the Kafka server

We needs 2 steps:

1.1 Start the ZooKeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

1	bin/zookeeper-server-start.sh config/zookeeper.properties

1.2. Start the Kafka server

bin/kafka-server-start.sh config/server.properties

1	bin/kafka-server-start.sh config/server.properties

2. List all topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

1	bin/kafka-topics.sh --zookeeper localhost:2181 --list

The output in my console:

hello-topic my-replicated-topic

1 2	hello-topic my-replicated-topic

3. Create a topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-topic

1	bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello-topic

The output is:

Created topic "hello-topic".

1	Created topic "hello-topic".

Above command will create a “hello-topic“, with replication-factor = 1 and the number of partitions is 1. Note that the replication-factor controls how many servers will replicate each message that is written; therefore, it should be less than or equal the number of Kafka servers/brokers.

4. Describe a topic

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello-topic

1	bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello-topic

The output is as below:

Topic:hello-topic PartitionCount:1 ReplicationFactor:1 Configs: Topic: hello-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

1 2	Topic:hello-topic       PartitionCount:1        ReplicationFactor:1     Configs:         Topic: hello-topic      Partition: 0    Leader: 0       Replicas: 0     Isr: 0

5. Publish messages to a topic

One of the most interesting command.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic

1	bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic

After this command, you can add any messages to the console, line by line. For ex:

topic Hello Halo Salut

1 2 3 4

topic Hello Halo Salut

 You can stop the console consumer via Ctrl-C

6. Consume messages

Below command will consume messages from the topic: hello-topic

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic

1	bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic

The output on my console is:

Hello Halo Salut

1 2 3

Hello Halo Salut

You can stop the console consumer via Ctrl-C

7. Alter Apache Kafka Topics

7.1. Add more partitions to the topic

Below command will add 10 more partitions to the hello-topic topic. Note that before, the topic has only 1 partition.

bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 11 --topic hello-topic

1	bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 11 --topic hello-topic

The output in my console:

WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected Adding partitions succeeded!

1 2	WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected Adding partitions succeeded!

7.2. Delete a topic

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic hello-topic

1	bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic hello-topic

7.3. Add configurations to the Kafka topic

The general syntax is:

bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic kafkatopic --config <key>=<value>

1	bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic kafkatopic --config <key>=<value>

For example,  below command will set the max message size = 128000 bytes for the hello-topic topic.

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --config max.message.bytes=128000

1	bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --config max.message.bytes=128000

Here is an example output:

WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases. Going forward, please use kafka-configs.sh for this functionality Updated config for topic "hello-topic".

1 2 3

WARNING: Altering topic configuration from this script has been deprecated and may be removed in future releases.          Going forward, please use kafka-configs.sh for this functionality Updated config for topic "hello-topic".

7.4. Add configurations to the Kafka topic

To remove above overridden configuration, we can use command:

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --delete-config max.message.bytes

1	bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic hello-topic --delete-config max.message.bytes

For the list of configurations, please reference Apache Kafka page

MQ

DEFINE QALIAS ('I1124.LOTUSNOTES.CUSTOMER.OAG.AQ.SUB') + 
       DESCR('I1124_LOTUSNOTES_CUSTOMER_OAG.msgflow') + 
       DEFPSIST(YES) + 
       TARGTYPE(TOPIC) +
       TARGET('I1124_LOTUSNOTES_CUSTOMER_OAG_Topic') + 
       REPLACE

DEFINE TOPIC ('I1124_LOTUSNOTES_CUSTOMER_OAG_Topic') +
       TOPICSTR('JCI/AE/EU/LOTUSNOTES/CUSTOMER/OAG') +
       DEFPSIST (YES) +
       DESCR ('I1124.LOTUSNOTES.CUSTOMER.OAG.AQ.SUB') +
       PUBSCOPE (ASPARENT) +
       SUB (ASPARENT) + 
       SUBSCOPE (ASPARENT) +
       WILDCARD (PASSTHRU) +
       REPLACE

DEFINE SUB(I1124_LOTUSNOTES_CUSTOMER_OAG) TOPICSTR('JCI/AE/EU/LOTUSNOTES/CUSTOMER/OAG') TOPICOBJ(SYSTEM.BROKER.DEFAULT.STREAM) DEST(I1124.LOTUSNOTES.CUSTOMER.OAG.SUB) DESTQMGR(MDBKRD07) PSPROP(RFH2)