Simple kafka producer and consumer

If you’ve read the nomenclature document, you will know that a producer is a program that writes data to a kafka topic, while a consumer is something that reads data from a Kafka topic.

This document will show you how to use Kafka provided simple console producer and consumer write and read data from Kafka.

Requirements

You will be using the kafka command line utilies for these actions. This is different than the AWS CLI tools (ie: aws kafka …).

Producer

It can be handy to have 2 sessions open while you do this - one with the producer, one with the consumer, so you can watch as data transits Kafka.

TIP: You can split your screen if you’re using iTerm by hitting command-d.

split-screen

Plaintext

To write data to a kafka topic, run the following. It will accept data from STDIN - you can hit ctrl-d to exit out when you are done.

./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic

Example:

./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
test
test2
test3
test4

CTRL-D

Consumer

Plaintext

To read data from a Kafka topic, you can do the following. To end, hit CTRL-C

./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic

Example:

./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
test1
test2
test3
test4

CTRL-C

If you connect to a busy topic you are going to see a lot of data - if you aren’t sure, you can pass the --max-messages parameter to limit the number of messages.

By default, the console-consumer will behave like tail - that is it will connect to the topic and print out all new messages. If you want it to start from the first available offset, you can add --offset earliest or --from-beginning

If you add --help to your command you can see there are a large number of options you can specifiy - the console consumer is simple, but can be very powerful, especially when troubleshooting.