If you’ve read the nomenclature document, you will know that a producer is a program that writes data to a kafka topic, while a consumer is something that reads data from a Kafka topic.
This document will show you how to use Kafka provided simple console producer and consumer write and read data from Kafka.
You will be using the kafka command line utilies for these actions. This is different than the AWS CLI tools (ie:
aws kafka …).
It can be handy to have 2 sessions open while you do this - one with the producer, one with the consumer, so you can watch as data transits Kafka.
TIP: You can split your screen if you’re using iTerm by hitting command-d.
To write data to a kafka topic, run the following. It will accept data from STDIN - you can hit
ctrl-d to exit out when you are done.
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic test test2 test3 test4 CTRL-D
To read data from a Kafka topic, you can do the following. To end, hit CTRL-C
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic test1 test2 test3 test4 CTRL-C
If you connect to a busy topic you are going to see a lot of data - if you aren’t sure, you can pass the
--max-messages parameter to limit the number of messages.
By default, the console-consumer will behave like
tail - that is it will connect to the topic and print out all new messages. If you want it to start from the first available offset, you can add
--offset earliest or
If you add
--help to your command you can see there are a large number of options you can specifiy - the console consumer is simple, but can be very powerful, especially when troubleshooting.