Go to the Cloudformation console, click on the stack you just created, click on the Outputs tab. Then scroll down and copy the value for the key KafkaClientEC2InstanceSsh.
SSH into the KafkaClientEC2Instance instance. Open a terminal on Mac or a putty terminal on Windows and paste the command you copied from the Outputs section above.
Go to /tmp/kafka directory and update the producer.properties_msk and schema-registry.properties files.
producer.properties_msk:
cd /tmp/kafka
nano producer.properties_msk
schema-registry.properties:
cd /tmp/kafka
nano schema-registry.properties
Start Schema Registry service
sudo systemctl start confluent-schema-registry
sudo systemctl status confluent-schema-registry
This is the expected output from running these commands.
Create the ExampleTopic topic in the MSK Kafka cluster.
/home/ec2-user/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper connect string you copied in the Setup** section> --replication-factor 3 --partitions 3 --topic ExampleTopic
Create the output topics in the MSK Kafka cluster where the Kinesis Data Analytics Apache Flink application would send the clickstream analytics to.
/home/ec2-user/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper connect string you copied in the Setup section> --replication-factor 3 --partitions 3 --topic Departments_Agg
/home/ec2-user/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper connect string you copied in the Setup section> --replication-factor 3 --partitions 3 --topic ClickEvents_UserId_Agg_Result
/home/ec2-user/kafka/bin/kafka-topics.sh --create --zookeeper <zookeeper connect string you copied in the Setup section> --replication-factor 3 --partitions 3 --topic User_Sessions_Aggregates_With_Order_Checkout
Run the KafkaClickstreamClient-1.0-SNAPSHOT.jar program. This is a mock Clickstream producer. Please read more about the producer at Producer.
Parameters
cd /tmp/kafka
java -jar KafkaClickstreamClient-1.0-SNAPSHOT.jar -t ExampleTopic -pfp /tmp/kafka/producer.properties_msk -nt 8 -rf 1800
This command will run the producer for 30 minutes.