Cluster creation with the Console

Creating an Amazon MSK cluster is quick and easy using the Console. A few steps and you’re off!

Create the cluster

Note that it can take some time to get the cluster up and running (15+ minutes).

  1. Sign into the AWS Console in the account you want to create your cluster in
  2. Browse to the MSK create cluster wizard to start the creation
  3. Select Create Cluster with custom settings
  4. Enter the cluster name - “MSKWorkshopCluster”
  5. Select the version of Kafka you want to run on the cluster (ex: 2.3.1)

Configuration Section

We are going to create a custom configuration that the cluster will run. We will enable the following features:

  • auto.create.topics.enable - allow topics to be created automatically by producers and consumers. This is not typically enabled in a production cluster, but it is handy for development and testing to lower the operational overhead
  • delete.topic.enable - enables topic deletion on the server. If topic deletion is not enabled, you cannot delete topics. You likely want to turn this on on all clusters you build unless you have a specific need not to.
  • log.retention.hours - we will set this to 8 hours for the lab. Note that this is the default configuration, it can still be overridden at the topic level

Create Configuration Object

  1. Select Use a custom configuration
  2. Click on ‘Create configuration’
  3. Give your configuration a name - WorkshopMSKConfig
  4. Add a description for the config - MSK workshop - Auto topic creation; topic deletion; 8hrs retention
  5. Under Configuration Properties, ensure the following options are set/overridden - you can leave the Version as None

Note that there is a default configuration that is provided - you can add new lines for new settings, and modify settings inline otherwise.

auto.create.topics.enable=true
delete.topic.enable=true
log.retention.hours=8

This includes some default configurations behind the scenes.


  1. Click Create - this will take you to the Cluster Configurations page. You can close this browser tab now
  2. Back in the Cluster Creation workflow, you can hit the refresh icon beside the cluster configuration box (the circle icon) then select the configuration you created

Note: If you want to see what configurations may already exist and what is contained in them, you can browse to https://console.aws.amazon.com/msk/home?region=us-east-1#/configurations

Networking Section

  1. Select the VPC you want to deploy your cluster in (AWSKafkaTutorialVPC if you created the VPC using our provided CloudFormation)
  2. Select the number of availability zones you want to deploy in to (3)
  3. Select us-east-1a for the first Availability Zone (AZ), then the subnet (PrivateSubnetMSKOne)
  4. Select us-east-1b for the second AZ, and the appropriate subnet (PrivateSubnetMSKTwo)
  5. Select us-east-1c for the third AZ, and the appropriate subnet (PrivateSubnetMSKThree)

Brokers

  1. Select kafka.m5.large as the Broker Instance Type

  2. Enter 1 for the number of brokers per AZ

Tags

  1. Under key enter Name and in value MSKLabCluster

Storage

  1. Enter 100 GiB

Encryption

Tip: You cannot enable encryption on an already created cluster, nor can you turn it off on a cluster configured with encryption, so plan your use carefully to avoid rebuilding to change these settings.

  1. Select ‘Enable Encryption within the cluster’

    Note that this can impact the performance of the cluster in production. If you don’t need this level of encryption consider leaving it off.

  2. Select Both TLS encrypted and plaintext traffic allowed. This will enable 2 different service ports on the cluster (9092 and 9094). You will be able to communicate in both an encrypted and unencrypted manner - choose based on your data needs. For this workshop we will experiment with both, but you should choose what fits your production environment best.

  3. Select Use AWS managed CMK. This means Amazon will manage the encryption key for you.

Authentication

  1. Leave Enable TLS client authentication blank. This feature as will be explored in other labs.

Monitoring

There are 2 types of monitoring available for Amazon MSK - Cloudwatch monitoring which is available in 3 flavours (Basic, Enhanced broker-level, Enhanced topic-level), as well as the Open Monitoring with Prometheus. We will use both.

  1. Select Enhanced topic-level monitoring. This will enable collection of metrics from each broker at the topic level. This generates more metrics and incurs additional costs, but will also let you troubleshoot and understand your traffic better.

  2. Select Enable open monitoring with Prometheus`

Broker Log Delivery

We are going to configure the cluster to send any broker logs to Cloudwatch Logs, which will provide us with an easy way to work with the logss

  1. Click Deliver to Amazon CloudWatch Logs

This will expand a new section where you will enter the ARN for the log group. To do this we will need to create a log group first!

  1. Click on visit Amazon CloudWatch Logs console - this will open a new tab in the CloudWatch Logs console
  2. Click on Create Log Group in the top right corner
  3. Enter your log group name - MSKClusterLogs and click Create
  4. You can now close this tab

You should be back in your MSK Cluster Creation window

  1. Click Browse button, and then check the circle beside MSKClusterLogs and then click Choose

Advanced Settings

In advanced settings, you can customize the security groups assocated with your cluster. In the prep step, we created a security group called MSK Service and we will attach that as well.

  1. Select Customize Settings, then in the drop down box select the MSKWorkshopKafkaService security group

And off we go!

Click Create Cluster - voila! Your cluster is being built. This can take 10-15 minutes, so it’s a good time to grab a coffee, read ahead in the lab, or explore the MSK Documentation


References