Setup

Run the CloudFormation template

  • Right click on the following link and open it in a new tab to execute the CloudFormation template.


msk connect launch stack screenshot

  • Click Next.

  • Choose a password for your Amazon Aurora MySQL Database.

msk connect launch stack screenshot

  • Click Next

  • Click Next on the next page.

  • Scroll down, check the checkbox next to I acknowledge that AWS CloudFormation might create IAM resources with custom names in the Capabilities section and click on Create stack.


It could take up to 30 minutes for the stack to run. Once the status of the stack changes to CREATE_COMPLETE, the stack is done creating. Please wait for the stack to complete and then proceed further.

  • Go to the Amazon MSK console. Click on the MSK cluster that was created by CloudFormation (default msk-connect-lab)

  • Click on View client information on the top right side of the page under Cluster summary.


  • Click on Copy under Bootstrap servers and paste it in a notepad application.

  • Click on Copy under Zookeeper connect and paste it in a notepad application. Click on Done.

  • Go to Amazon EC2 Console. Select the KafkaClientInstance instance by checking the checkbox. Click on the Connect button.

  • From the available options, choose Session Manager. Wait a few seconds for the Connect button to change to active from grayed out.


  • Click on Connect button, this will open a new browser window with a browser based terminal.

  • Run the command below by replacing <Your Bootstrap servers address> with the copied values.

sudo -u ec2-user -i
export MSK_BOOTSTRAP_ADDRESS=<Your Bootstrap servers address>
  • Copy and paste the bash script below in the opened terminal. This will set up a few of the stack information - such as RDS Aurora Endpoint url, AWS Region, Account Id - in the EC2 instance’s bash_profile.
echo "export MSK_BOOTSTRAP_ADDRESS=${MSK_BOOTSTRAP_ADDRESS}" | tee -a ~/.bash_profile

export AWS_REGION=$(curl -s 169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.region')
export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
test -n "$AWS_REGION" && echo AWS_REGION is "$AWS_REGION" || echo AWS_REGION is not set
export RDS_AURORA_ENDPOINT=$(aws rds describe-db-instances --region ${AWS_REGION} | jq -r '.DBInstances[] | select(.DBName == "salesdb") | .Endpoint.Address')
echo "export RDS_AURORA_ENDPOINT=${RDS_AURORA_ENDPOINT}" | tee -a ~/.bash_profile
echo "export ACCOUNT_ID=${ACCOUNT_ID}" | tee -a ~/.bash_profile
echo "export AWS_REGION=${AWS_REGION}" | tee -a ~/.bash_profile
  • Copy the Aurora RDS Endpoint URL from the output and paste it in your notepad application.

Finally, you need to configure your MSK cluster, to auto-create topics, per each table in our salesdb database.

To update the configuration of a cluster, make sure that the cluster is in the ACTIVE state.

  • Run the following command to “Create cluster configuration”:
sudo -u ec2-user -i
mkdir kafka-configuration && cd kafka-configuration
cat <<EOF > configuration.txt 
auto.create.topics.enable=true
EOF

aws kafka create-configuration \
    --name "CustomConfiguration" \
    --description "Topic auto-creation enabled" \
    --kafka-versions "2.6.2" \
    --region $AWS_REGION \
    --server-properties file://configuration.txt
  • Navigate to the clusters section, find MSKCluster-msk-connect-lab, and click on the cluster name.

  • Find the Cluster Configuration tab, and click on the Edit:


  • From the Cluster Configuration dropdown choose the CustomConfiguration.

  • From the Configuration revision dropdown choose Revision 1.

  • Click Save changes button.

  • Cluster status will change to Updating. Wait a few minutes until the status changes to Active again.