Setup Monitoring

Steps we will perform in this section

  • Setup Grafana and Prometheus used for monitoring.

Monitoring MirrorMaker 2 on Kafka Connect

Note: If you’re connected to VPN, it might be advisable to disconnect before you do these steps, so that the VPN does not block access.

  • Go back to the AWS Cloud9 console. Click on MSKClient-Cloud9EC2Bastion.

  • Click on Open IDE (or go back to the browser tab if alreay open).


  • Rick click in the space between the bash pane and the menu bar and click on Split pane in Two Columns. Now you have 3 terminals.


  • In the new pane created in the middle, click on the small green circle with a + and click on New Terminal. Expand the terminal so you have more space.


  • Download the monitoring setup script from s3 and make it executable

    aws s3 cp s3://reinvent2019-msk-liftandshift/setup-monitoring.sh /home/ec2-user/environment && chmod +x setup-monitoring.sh
    

  • Run the monitoring setup script
    Note: The stack name is the name of the Amazon MSK stack that you created in the Lab Setup section and the region is the AWS region you created the stack.
    Note: The IP addess is the IP that will be given access to the Grafana and Prometheus dashboard urls.

    ./setup-monitoring.sh <stackName> <region> <IPAddress of laptop/32> <filesystem location to install grafana and prometheus>
    

  • In the Output check to make sure both the grafana-server and the prometheus services are running.


  • Run the following commands to get the url of grafana (first one) and prometheus (second one) servers

    echo "http://$(curl -X GET http://169.254.169.254/latest/meta-data/public-hostname):3000"
    echo "http://$(curl -X GET http://169.254.169.254/latest/meta-data/public-hostname):9090"
    
  • Go to a browser window and paste in the url for grafana. Then in the login screen type in admin for username and admin for password. Click on Log in.


  • You will be asked to change the password. Please provide a new password and note it down.


  • You will now see the home page. On the left panel, click on the gear icon and click on Data Sources.


  • A prometheus data source is already configured for you. Click on prometheus1.


  • Scroll down and click on Save & Test.


  • You should get a message saying Data souce is working.


  • Click on the Dashboard icon in the left panel. Then click on Manage.


  • In the bottom panel, click on MirrorMaker2. Then click on MM2.


  • The pre-created dashboard will launch. However, you won’t see any data yet. As you start the Clickstream producer, consumer and MM2, this dashboard will start getting data. Come back to the dashboard and periodically hit the refresh button in the top right corner or adjust the time range to show data in the graph.


  • The most important metric here is the MM2-Consumer-Lag. It indicates how far behind the end of the log for each partition in the ExampleTopic in the source Amazon MSK cluster, the MM2 consumer is. In this graph, I had started the producer first and some time later started MM2. You can see that the lag was pretty high but then dropped to zero as the consumers caught up. Before cutting over the producer, check to make sure that the lag is zero to avoid ordering issues.

  • Go back to the browser window with the Cloud9 IDE and x out of the new terminal that you created in the middle.