If you’ve read the nomenclature document, you will know that a producer is a program that writes data to a kafka topic, while a consumer is something that reads data from a Kafka topic.
This document will show you how to use Kafka provided simple console producer and consumer write and read data from Kafka.
$MYBROKERS
You will be using the kafka command line utilies for these actions. This is different than the AWS CLI tools (ie: aws kafka
…).
It can be handy to have 2 sessions open while you do this - one with the producer, one with the consumer, so you can watch as data transits Kafka.
TIP: You can split your screen if you’re using iTerm by hitting command-d.
To write data to a kafka topic, run the following. It will accept data from STDIN - you can hit ctrl-d
to exit out when you are done.
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
Example:
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
test
test2
test3
test4
CTRL-D
To read data from a Kafka topic, you can do the following. To end, hit CTRL-C
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
Example:
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
test1
test2
test3
test4
CTRL-C
If you connect to a busy topic you are going to see a lot of data - if you aren’t sure, you can pass the --max-messages
parameter to limit the number of messages.
By default, the console-consumer will behave like tail
- that is it will connect to the topic and print out all new messages. If you want it to start from the first available offset, you can add --offset earliest
or --from-beginning
If you add --help
to your command you can see there are a large number of options you can specifiy - the console consumer is simple, but can be very powerful, especially when troubleshooting.