Creating an Amazon MSK cluster is quick and easy using the Console. A few steps and you’re off!
Note that it can take some time to get the cluster up and running (15+ minutes).
Create Cluster with custom settings
We are going to create a custom configuration that the cluster will run. We will enable the following features:
8hours for the lab. Note that this is the default configuration, it can still be overridden at the topic level
Use a custom configuration
MSK workshop - Auto topic creation; topic deletion; 8hrs retention
Note that there is a default configuration that is provided - you can add new lines for new settings, and modify settings inline otherwise.
auto.create.topics.enable=true delete.topic.enable=true log.retention.hours=8
This includes some default configurations behind the scenes.
Note: If you want to see what configurations may already exist and what is contained in them, you can browse to https://console.aws.amazon.com/msk/home?region=us-east-1#/configurations
AWSKafkaTutorialVPCif you created the VPC using our provided CloudFormation)
us-east-1afor the first Availability Zone (AZ), then the subnet (
us-east-1bfor the second AZ, and the appropriate subnet (
us-east-1cfor the third AZ, and the appropriate subnet (
Custom Settings, then in the drop down box select the
MSKWorkshop-KafkaService security group we created in Step 2 of the previous Preparation steps.
kafka.m5.large as the Broker Instance Type
1 for the number of brokers per AZ
Tip: You cannot enable encryption on an already created cluster, nor can you turn it off on a cluster configured with encryption, so plan your use carefully to avoid rebuilding to change these settings.
Select ‘Enable Encryption within the cluster’
Note that this can impact the performance of the cluster in production. If you don’t need this level of encryption consider leaving it off.
Both TLS encrypted and plaintext traffic allowed. This will enable 2 different service ports on the cluster (9092 and 9094). You will be able to communicate in both an encrypted and unencrypted manner - choose based on your data needs. For this workshop we will experiment with both, but you should choose what fits your production environment best.
Use AWS managed CMK. This means Amazon will manage the encryption key for you.
Enable TLS client authenticationblank. This feature as will be explored in other labs.
There are 2 types of monitoring available for Amazon MSK - Cloudwatch monitoring which is available in 3 flavours (Basic, Enhanced broker-level, Enhanced topic-level), as well as the Open Monitoring with Prometheus. We will use both.
Enhanced topic-level monitoring. This will enable collection of metrics from each broker at the topic level. This generates more metrics and incurs additional costs, but will also let you troubleshoot and understand your traffic better.
Select Enable open monitoring with Prometheus`
We are going to configure the cluster to send any broker logs to Cloudwatch Logs, which will provide us with an easy way to work with the logss
Deliver to Amazon CloudWatch Logs
This will expand a new section where you will enter the ARN for the log group. To do this we will need to create a log group first!
visit Amazon CloudWatch Logs console- this will open a new tab in the CloudWatch Logs console
Create Log Groupin the top right corner
You should be back in your MSK Cluster Creation window
Browsebutton, and then check the circle beside
MSKClusterLogsand then click
Nameand in value
Click Create Cluster - voila! Your cluster is being built. This can take 10-15 minutes, so it’s a good time to grab a coffee, read ahead in the lab, or explore the MSK Documentation