Open Monitoring Preparation

  • This Lab requires you to have completed the Cluster Creation Lab. If you haven’t completed it, please complete it and then come back to this lab.

In order to use Open Monitoring with Prometheus, you need to have completed the steps in this preparation guide.

OPTIONAL - Step 1 - Enable Open Monitoring on existing cluster

The critical step to using Open Monitoring is that you need to enable it. It’s not enabled by default on all clusters, so if you didn’t enable it when you created your cluster, you’ll have to turn it on.

Any clusters created as part of this workshop will have Open Monitoring enabled, but if you’re using this against an existing cluster, this section will help ensure you’re ready.

  1. Sign in to the account where the Amazon MSK cluster is you want to monitor

  2. Open the Amazon MSK Cluster Console

  3. Click on the Amazon MSK cluster you want to enable monitoring on

  4. Scroll to the Monitoring section

  5. If you have already enabled Open Monitoring, you will see a screen similar to this - you can continue to the next preparation step if so:

  6. If you haven’t enabled Open Monitoring, you will see a screen similar to this:

  7. Enable Open Monitoring

Since it’s not enabled, lets turn it on!

  • Click on the ‘Edit’ button to the right of the Monitoring header
  • Select the Enable open monitoring with Prometheus checkbox
  • Select the JMX Exporter and the Node Exporter checkboxes
  • Click Save Changes

This will trigger a configuration update on the cluster hat will take a couple minutes to apply. You will see a blue bar indicated the operation is in progress. When it is complete, the bar will turn green to indicate the operation is complete.


You are set! Now you can move on to Step 2 and make the monitoring ports accessible.

Step 2 - Create Security group rules to allow access to monitoring

  1. Create a new SG called MSK_Monitoring with no rules

    1. Open the EC2 service, then select Security Groups in the left navigation pane
    2. Click Create Security Group at the top of the screen
    3. Enter the name MSK_Monitoring
    4. Description is Access to MSK monitoring from monitoring services
    5. Ensure that you select the VPC your Amazon MSK Cluster is deployed to (AWSKafkaTutorialVPC)
    6. Click Create, having added no rules

Example:

  1. Modify the MSK Workshop Service SG to include 2 new rules

  • Open the EC2 service, then go to Security Groups

  • Click on the MSKWorkshop-KafkaService then in the bottom pane click Edit and add the following rule

    • Type: Custom TCP
    • Port range: 11001-11002
    • Source: MSK_Monitoring security group
    • Description: Prometheus monitoring

Note: In the Source field, simply start typing MSK_monitoring and it will populate with the security group ID.

Example:

  1. Attach the MSK_Monitoring SG to your KafkaClientInstance instance and your Cloud9 instance

    1. Open the EC2 console and go to Instances
    2. Select your KafkaClientInstance host
    3. Select Actions from the top bar
    4. Click on Networking then Change Security Groups
    5. Check the box next to MSK_Monitoring and press Assign Security Groups in the bottom right corner
    6. Do the same for the Cloud9 instance (named aws-cloud9-msklab...)

This will be used shortly