Amazon Cloudwatch Dashboard

  1. Open the Amazon CloudWatch console at https://console.aws.amazon.com/cloudwatch/

  2. Select Dashboards in the Navigation pane (left side)

  3. Select Create dashboard and enter the name MSK-Monitoring

  4. Select Create Dashboard

  5. On the Add to this Dashboard dialogue, you can choose from various types of graphs. For now, we will select the Line type and click Configure

Lets graph some metrics!

  1. Click on AWS/Kafka, then Broker ID, Cluster Name

  2. In the search box enter the name of your Amazon MSK Cluster (ex: MSKCluster) and press enter. This will limit metrics to that cluster

  3. Enter CpuUser in the search box and press enter. This will show you only the CpuUser metric for the cluster.

  4. On the left side beside Broker ID click the empty checkbox to select all the metrics listed

  5. This will update the graph with data from each brokers CPU

  6. At the top beside Untitled Graph click the pencil and give your graph a name (ex: MSK Cluser CPU (User) utilization)

  7. Click Create widget in the bottom right corner

  8. Your dashboard will be updated now with a graph containing the CPU utilization of all your brokers! Lets do the same for a few more metrics:

  • GlobalPartitionCount - (found under AWS/Kafka -> Cluster Name) - after you’ve graphed this one, click ‘Graphed Metrics’ and change the Statistic to Maximum
  • OfflinePartitionsCount - (found under AWS/KAfka -> Cluster Name) - set this to Maximum as well
  • GlobalTopicCount - (found under AWS/KAfka -> Cluster Name) - set this to Maximum as well
  • ZooKeeperRequestLatencyMsMean - (found under AWS/KAfka -> Cluster Name) - set this to p99
  • KafkaDataLogsDiskUsed - (found under AWS/Kafka -> Broker ID, Cluster Name) - set this to Maximum
  • MessagesInPerSec - (found under AWS/Kafka -> Broker ID, Cluster Name) - set this to Maximum
  • UnderReplicatedPartitions - (found under AWS/Kafka -> Broker ID, Cluster Name) - set this to Maximum
  • BytesOutPerSec - (found under AWS/Kafka -> Broker ID, Cluster Name) - set this to Maximum
  • BytesInPerSec - (found under AWS/Kafka -> Broker ID, Cluster Name) - set this to Maximum

You can add as many graphs as you want, and create multiple dashboards. Try different features, different options, different graph types, etc.

  1. Remember to hit Save Dashboard before you navigate away from the page or your work will be lost!

Resources