Create Kibana Dashboard

Create Kibana Dashboard

  • The Amazon Elasticsearch cluster is provisioned in a VPC. Hence, the Amazon Elasticsearch endpoint and the Kibana endpoint are not available over the internet. In order to access the endpoints, we have to create a ssh tunnel and do local port forwarding.

    • For Mac:
      • In a terminal, go to ~/.ssh directory, edit the config file and add the following configuration -

        Host estunnel  
         HostName <from the CloudFormation Outputs section copy the value of the key KafkaClientEC2InstancePublicDNS >  
         User ec2-user  
         IdentitiesOnly yes  
         IdentityFile <the location of the ssh pem file created in the Prerequisites section>  
         LocalForward 9200 <from the CloudFormation Outputs section copy the value of the key ElasticsearchEndpoint without the "https://">:443
        
        cd ~/.ssh
        nano config
        ssh estunnel
        
    • For Windows:
      • Open PuTTY.
      • For Host Name, put in ec2-user@<copy the value of KafkaClientEC2InstancePublicDNS from the Outputs tab of the Cloudformation stack>.

      • For Port, put in 22. Then expand SSH in the left pane and click on Auth.


      • Click on the checkbox for Allow agent forwarding.

      • Click on Browse and select the ppk file you created in the Prerequisites section. Then click on Tunnels.


      • Click on the checkbox for Local ports accept connections from other hosts.

      • For Source port, put in 9092.

      • For Destination, put in <copy the value of ElasticsearchEndpoint without “https://” from the Outputs tab of the Cloudformation stack>:443.

      • Click on Add.


      • This will add the Destination and Port to the Forwarded ports box. Click on Open.


      • This will open an ssh tunnel.


  • Open a browser and access Kibana.

    https://127.0.0.1:9200/_plugin/kibana
    
  • In the left pane, scroll down and click on Management

    Kibana Management
  • Click on Index Patterns and then click on Create index pattern

    Index Patterns
  • Type in departments_count under Index pattern and click on Next step

    Create Index Pattern
  • In the dropdown next to Time Filter field name, select windowEndTime. Then click on Create index pattern

    Create Index Pattern
  • The Index pattern will be created.

    Create Index Pattern
  • Repeat the process and create two more index patterns for:

    • user_session_counts

      Create Index Pattern
      Create Index Pattern
      Create Index Pattern
    • user_session_details

      Create Index Pattern
      Create Index Pattern
      Create Index Pattern
  • Click on Visualize in the left pane.

    Visualize
  • Click on Create a visualization.

    Visualize
  • Select Line as the Visualization type

    Visualization Type
  • In the next screen, select departments_count

    Departments Count
  • Configure the departments_count visualization as shown in the picture below and click on the play button at the top of the pane.

    Departments Count Configuration
  • The chart will update. If you see no data, adjust the Time Range in the top right of the page to reflect the time the producer produced events into Amazon MSK.

    Departments Count Chart
  • Save the visualization by clicking on Save at the top of the screen.

  • Repeat the process for two more remaining index patterns. Remember to save the visualizations so you can add them to a dashboard later.

    User Session Counts Chart
    User Session Counts With Order Checkout Chart
    Percent User Session Counts With Buy Chart
    Event Counts for User Sessions With Buy Chart
  • Scroll up and click on Dashboard. Then, click on Create new dashboard.

    Dashboard
  • Click on Add on the next screen.

    Dashboard
  • Click on all the visualizations you would like to add to the dashboard and X out of the window.

  • Save the dashboard.

    Dashboard
  • Your dashboard is ready!

    Dashboard
  • Observe as new events are generated and processed, the graphs in the dashboard get updated in real-time.

You have reached the end of this lab. Congratulations!!

Please go to the Clean up section and clean up the resources in your account.