Lab Setup

Steps we will perform in this section

Create or Confirm Environment

As previously noted, this lab can be run in your own AWS environment or in an AWS environment provided to you via EventEngine. EventEngine is used to procure environments for interactive and guided workshops. Please note these instructions cover both scenarios, so please pay close attention to which scenario is applicable to you.

If you are running this lab from EventEngine with AWS, you may skip this section and move to Confirm Environment.

If you are running this lab in your own AWS environment, we need to run a Cloud Formation template to create your lab environment.

First, from AWS Console, please ensure you are using the appropriate region. Please choose from one of the following.

# Region Name Code
1 US East (N. Virginia) us-east-1
2 US West (Oregon) us-west-2
3 EU (Ireland) eu-west-1

Next, right click on the following Launch Stack button and open it in new tab to execute the CloudFormation template. You can download the CloudFormation template here.

After opening, click the orange “Next” button in the lower right hand corner.

Then, the next screen of the template will ask for various inputs–

  • Set the Stack Name to “MSKMM”.
  • Set PCAARNDestinationMSKCluster and PCAARNSourceMSKCluster fields to null
  • Set both TLS drop down options both to false

Screenshot Example

Once complete, you may move to the next section. At time of this writing, it takes about 15-20 minutes to complete.

Confirm Your Environment

In order to complete this lab, we need a source and destination MSK cluster. We can confirm our environment by going to MSK Service in the AWS Console and seeing two MSK Clusters. For example, MSKCluster1-MSKMM and MKSCluster2-MSKMM are shown in the following, but your cluster names will differ if running this lab in Event Engine.


Create the Cloud9 Bastion environment and the Apache Kafka Client EC2 instances

Make sure you have either created or downloaded an EC2 KeyPair before proceeding. If you are running in Event Engine, you can download your EC2 KeyPair from your Team Dashboard. If running in your environment, you can create an EC2 KeyPair as shown in the Prerequisites section.

In this section, we will create EC2 instances with Apache Kafka and a Cloud9 Bastion host, so we can connect and administer to connect to our Cloud9 Bastion host with our web browser. Then, we can ssh to our EC2 instance(s) to interact with our MSK Clusters from Apache Kafka command line tools, setup Kafka Connect and MirrorMaker.

If you are running this in EventEngine, please see Required Event Engine Variables before proceeding.

  • Right click on Launch Stack and open it in a new tab to execute the CloudFormation template. You can download the CloudFormation template here.


After opening from Launch Stack button above, click the orange “Next” button in the lower right hand corner. Then you will see a screen similar to the following:


  • Set “Stack name” to “MSKClients”.

  • Choose the EC2 KeyPair that you created in the Prerequisites or downloaded from the Event Engine Team Dashboard. If running in Event Engine, set to “ee-default-keypair.pem”

  • MSKStack should be set to “MSKMM” if you are running this lab in your environment. We set it to “MSKMM” in the Create or Confirm Environment section above. If you’re running this lab in EventEngine, set to your unique “Stack name” if running via Event Engine. (For Event Engine, see Required Event Engine Variables as previously noted.)

  • VPCStack should be set to VPCStackName from your MSK cluster creation Outputs and depends on if you are running in your own environment or via Event Engine. (Again, for Event Engine, see Required Event Engine Variables as previously noted. For example, it would be “MSK-MSKVPCStack-130Q9QPVFO0H1” based on the following example output from MSK Cluster Output:


  • Click Next.

  • And again, click Next on the next page because the default settings are fine.

  • Scroll down, check the one or more Acknowledgement checkboxes. The amount of checkboxes will vary if running in EventEngine vs. your own AWS account. The example below shows checkboxes for I acknowledge that AWS CloudFormation might create IAM resources with custom names and I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND in the Capabilities section and click on Create stack. (Again, please note your “Capabilities” section might appear different than screenshot below, but that’s ok. The important thing is to check any Acknowledgement checkboxes before clicking Create stack)


    This creation process usually takes 5 minutes and to recap, this CloudFormation stack creates:

    1. A Cloud9 environment to be used as a “jump” box or also known as a bastion host.
    2. Two EC2 instances with Apache Kafka downloaded and helpful scripts used later in the lab.

Setup SSH keys in the Cloud9 environments

  • Go to the AWS Cloud9 console.

  • Click on MSKClient-Cloud9EC2Bastion and then click Open IDE.


  • In the Getting started section, click on Upload Files…


  • Click on Select files. Pick the EC2 pem file that you downloaded in Event Engine Team Dashboard or created in the Prerequisites section. Click Open. The file will be copied to the /home/ec2-user/environment dir and will also be visible in the left pane.


  • Go to the bash pane at the bottom and type in the following commands to setup the ssh environment so that you can access the Kafka Client EC2 instances. For example, run the following commands if you are running in EventEngine environment. If you are running in your own AWS environment, change the name ee-default-keypair.pem to the name of your .pem file.

    chmod 600 ee-default-keypair.pem
    eval `ssh-agent`
    ssh-add -k ee-default-keypair.pem