In order to use Open Monitoring with Prometheus, you need to have completed the steps in this preparation guide.
The critical step to using Open Monitoring is that you need to enable it. It’s not enabled by default on all clusters, so if you didn’t enable it when you created your cluster, you’ll have to turn it on.
Any clusters created as part of this workshop will have Open Monitoring enabled, but if you’re using this against an existing cluster, this section will help ensure you’re ready.
Sign in to the account where the Amazon MSK cluster is you want to monitor
Open the Amazon MSK Cluster Console
Click on the Amazon MSK cluster you want to enable monitoring on
Scroll to the Monitoring section
If you have already enabled Open Monitoring, you will see a screen similar to this - you can continue to the next preparation step if so:
If you haven’t enabled Open Monitoring, you will see a screen similar to this:
Enable Open Monitoring
Since it’s not enabled, lets turn it on!
Enable open monitoring with Prometheus
checkboxJMX Exporter
and the Node Exporter
checkboxesSave Changes
This will trigger a configuration update on the cluster hat will take a couple minutes to apply. You will see a blue bar indicated the operation is in progress. When it is complete, the bar will turn green to indicate the operation is complete.
You are set! Now you can move on to Step 2 and make the monitoring ports accessible.
Create a new SG called MSK_Monitoring
with no rules
Security Groups
in the left navigation paneCreate Security Group
at the top of the screenMSK_Monitoring
Access to MSK monitoring from monitoring services
AWSKafkaTutorialVPC
)Create
, having added no rulesExample:
MSK Workshop Service
SG to include 2 new rulesOpen the EC2 service, then go to Security Groups
Click on the MSKWorkshop-KafkaService
then in the bottom pane click Edit
and add the following rule
Note: In the Source
field, simply start typing MSK_monitoring
and it will populate with the security group ID.
Example:
Attach the MSK_Monitoring SG to your KafkaClientInstance instance and your Cloud9 instance
Actions
from the top barNetworking
then Change Security Groups
MSK_Monitoring
and press Assign Security Groups
in the bottom right corneraws-cloud9-msklab...
)This will be used shortly