Simple kafka producer and consumer
If you’ve read the nomenclature document, you will know that a producer is a program that writes data to a kafka topic, while a consumer is something that reads data from a Kafka topic.
This document will show you how to use Kafka provided simple console producer and consumer write and read data from Kafka.
Requirements
- Your kafka broker list as
$MYBROKERS - Connect to your KafkaClientInstance via SSH in Cloud9
You will be using the kafka command line utilies for these actions. This is different than the AWS CLI tools (ie: aws kafka …).
Producer
It can be handy to have 2 sessions open while you do this - one with the producer, one with the consumer, so you can watch as data transits Kafka.
TIP: You can split your screen if you’re using iTerm by hitting command-d.
Plaintext
To write data to a kafka topic, run the following. It will accept data from STDIN - you can hit ctrl-d to exit out when you are done.
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
Example:
./kafka-console-producer.sh --broker-list $MYBROKERS --topic ExampleTopic
test
test2
test3
test4
CTRL-D
Consumer
Plaintext
To read data from a Kafka topic, you can do the following. To end, hit CTRL-C
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
Example:
./kafka-console-consumer.sh --bootstrap-server $MYBROKERS --topic ExampleTopic
test1
test2
test3
test4
CTRL-C
If you connect to a busy topic you are going to see a lot of data - if you aren’t sure, you can pass the --max-messages parameter to limit the number of messages.
By default, the console-consumer will behave like tail - that is it will connect to the topic and print out all new messages. If you want it to start from the first available offset, you can add --offset earliest or --from-beginning
If you add --help to your command you can see there are a large number of options you can specifiy - the console consumer is simple, but can be very powerful, especially when troubleshooting.