Preparation

The following steps will prepare you for the creation of the Amazon MSK cluster in this lab.

Note that running this module will launch AWS resources that have an associated cost. If you are not running this lab as part of an Amazon MSK workshop using provided accounts, remember to clean up when you are done to keep from incurring ongoing charges for resources left running.

Step 1- Create Lab resources

If you want to keep your lab work separate or you’re using an existing account and don’t want to overlap with others, you can create a VPC and Subnets to work in. Follow these easy steps to get started:

  • Ensure that you’ve created an SSH key in your account

  • Use CloudFormation to launch an Amazon MSK focused VPC


    • Change the SSHLocation if you want to limit SSH access
    • Provide the SSH Key that you created in the prerequisites section
    • Click Next then Create Stack

    The stack creates:

    1. A VPC with 1 Public subnet and 3 Private subnets and the required plumbing including a NAT Gateway.
    2. A Cloud9 instance you will use as a Bastion
    3. 1 Apache KafkaClientInstance - an EC2 instance that has Apache Kafka, AWS CLI v1, AWS CLI v2 (aws2), jq, docker, an Apache Kafka Clickstream producer and schema registry service installed.
    4. A security group for the Apache Kafka client instance associated with the EC2 instance that needs to be given access to from the security group for the Amazon MSK cluster in this lab.

  • If you wish, you can download the CloudFormation template here

You can now proceed with the prep and lab. Ensure that you use the new VPC when you’re completing the lab, it won’t be the default so make sure you pick it from the drop downs presented.

Step 2 - Create an MSK security group

By default, the cluster will be attached to the ‘default’ security group, which allows all ports between all members of the group. This is fine for testing, but it’s not a best practice in production.

We need two security groups - one to attach to producers, consumers, and admin hosts, and the other to attach to the Amazon Amazon MSK cluster that references the first.

The CloudFormation template already created one of them - the “client” security group. Look at the Outputs tab of the CloudFormation stack created in the previous step and copy the value for the key KafkaClientEC2InstanceSecurityGroupId.

  • Create the security group for the Amazon MSK cluster.

    1. Click on Services in the top left corner of the console, and select EC2

    2. Select Security Groups in the left pane

    3. Click Create Security Group

    4. Fill out the form as follows:

Security group name: MSKWorkshop-KafkaService Description: Access to the Kafka service on the MSK cluster VPC: [select the VPC you are using for your lab (MSKVPC)]

  1. Create rules

    a. Click Add rule
    Use:

    • Type: Custom TCP
    • Protocol: TCP
    • Port range: 9092
    • Source: [paste the value of the KafkaClientEC2InstanceSecurityGroupId SG you copied in the previous step, from Cloudformation Outputs (msklab-KafkaClientInstance…)] Source2: Add the Cloud9 security group by typing cloud9 into the custom field - you’ll see something like aws-cloud9-msklab...)
    • Description: Plaintext Kafka

    c. Click Add Rule
    Use:

    • Type: Custom TCP
    • Protocol: TCP
    • Port range: 9094
    • Source: [paste the value of the KafkaClientEC2InstanceSecurityGroupId SG you copied in the previous step, from Cloudformation Outputs (msklab-KafkaClientInstance…)] Source2: Add the Cloud9 security group by typing cloud9 into the custom field - you’ll see something like aws-cloud9-msklab...)
    • Description: Encrypted Kafka

    e. Click Add Rule
    Use:

    • Type: Custom TCP
    • Protocol: TCP
    • Port range: 2181
    • Source: [paste the value of the KafkaClientEC2InstanceSecurityGroupId SG you copied in the previous step, from Cloudformation Outputs (msklab-KafkaClientInstance…)] Source2: Add the Cloud9 security group by typing cloud9 into the custom field - you’ll see something like aws-cloud9-msklab...)
    • Description: Zookeeper access

  2. Click Create

Example:


In the security group list, select the Group ID for the MSKWorkshopKafkaService SG, and copy it to your notepad/texteditor

You are done! Now you can move on to the lab.

Resources