In this lab we will look at the simple monitoring available through the Amazon MSK Service
Login to the Console of the account where your cluster is running and go to the Amazon MSK Service Console.
Click on the name of the Amazon MSK Cluster you are interested in monitoring
Click on the
You will see a simple dashboard that’s showing you metrics from your cluster:
Create CloudWatch alarm- let’s click that!
Click on ‘AWS/Kafka’ to explore the metrics available from your Amazon MSK Cluster
Broker ID, Cluster Name to get to the Disk space metrics.
Add to search
In the Search bar, enter
KafkaDataLogsDiskUsed to filter to the metric we want to alarm on. You can also find the metric in the list, select the down arrow and
Add to search
Check box beside Broker ID
1 - you will see a graph appear above, showing you the values of the metric for the given broker. We will use this to create your disk space alarm
Select Metricin the bottom right corner.
This will take you back to the Create Alarm wizard where you will now configure the rest of the alarm - thresholds, actions, and metadata.
Scroll down to
Conditions and select Static and Greater because we want to alarm on our metric (
KafkaDataLogsDiskUsed) being greater than 85%
Threshold Value enter
Select the down arrow beside
Datapoints to Alarm enter
2 in the first box, and
3 in the second - this indcates that we want 2 out of 3 polls (every 5 min) to be above 85% before alarming - this should help make sure the cluster is above 85% and didn’t just quickly spike over.
Missing Data Treatment as
Treat missing data as missing - this will not impact behvavior of the alarm
Next to configure the action to take
In the Configure Actions section, select
Create new topic and enter
msk_cloudwatch_alarm as the name below the radio box
Enter your email address in the
Email endpoints that will receive the notification
This step will create an SNS topic with your email address as the garget, and it will send a confirmation email to your address. Please confirm your email by clicking on the link.
Click ‘Next’ button in the bottom right to fill out the metadata for the alarm
Alarm Name enter
MSK Broker Disk Utilization - Broker 1
Alarm description enter
MSK broker data volume is over 85% utilized. Investigate and add capaicty if required.
Click ‘Next’ button in the bottom right to review your Alarm and activate it
Scroll through the summary of your alarm, and if everything looks correct, click
Congrats - you have an alarm for broker 1 disk space! But we want the other brokers in your cluster to be monitored too! So we will duplicate this alarm and tweak it to monitor the other brokers.
You will notice the the alarm will start in state
Insufficient data - this is because the alarm is still waiting for enough polls to pass by to establish a known state - then it will be marked
Click the checkbox beside
MSK Broker Disk utilization - Broker 1, then click
Broker ID change the value to
2 then scroll down and hit Next and Next
Name and Description change
Broker 1 to
Broker 2 and hit Next
Repeat the previous 4 steps for other brokers in the cluster. Otherwise, you can move on to the next exercise, or you can try creating additional alarms and explore other features..