The Amazon Elasticsearch cluster is provisioned in a VPC. Hence, the Amazon Elasticsearch endpoint and the Kibana endpoint are not available over the internet. In order to access the endpoints, we have to create a ssh tunnel and do local port forwarding.
In a terminal, go to ~/.ssh directory, edit the config file and add the following configuration -
Host estunnel HostName <from the CloudFormation Outputs section copy the value of the key KafkaClientEC2InstancePublicDNS > User ec2-user IdentitiesOnly yes IdentityFile <the location of the ssh pem file created in the Prerequisites section> LocalForward 9200 <from the CloudFormation Outputs section copy the value of the key ElasticsearchEndpoint without the "https://">:443
cd ~/.ssh nano config ssh estunnel
For Host Name, put in ec2-user@<copy the value of KafkaClientEC2InstancePublicDNS from the Outputs tab of the Cloudformation stack>.
For Port, put in 22. Then expand SSH in the left pane and click on Auth.
Click on the checkbox for Allow agent forwarding.
Click on Browse and select the ppk file you created in the Prerequisites section. Then click on Tunnels.
Click on the checkbox for Local ports accept connections from other hosts.
For Source port, put in 9092.
For Destination, put in <copy the value of ElasticsearchEndpoint without “https://” from the Outputs tab of the Cloudformation stack>:443.
Click on Add.
This will add the Destination and Port to the Forwarded ports box. Click on Open.
This will open an ssh tunnel.
Open a browser and access Kibana.
In the left pane, scroll down and click on Management
Click on Index Patterns and then click on Create index pattern
Type in departments_count under Index pattern and click on Next step
In the dropdown next to Time Filter field name, select windowEndTime. Then click on Create index pattern
The Index pattern will be created.
Repeat the process and create two more index patterns for:
Click on Visualize in the left pane.
Click on Create a visualization.
Select Line as the Visualization type
In the next screen, select departments_count
Configure the departments_count visualization as shown in the picture below and click on the play button at the top of the pane.
The chart will update. If you see no data, adjust the Time Range in the top right of the page to reflect the time the producer produced events into Amazon MSK.
Save the visualization by clicking on Save at the top of the screen.
Repeat the process for two more remaining index patterns. Remember to save the visualizations so you can add them to a dashboard later.
Scroll up and click on Dashboard. Then, click on Create new dashboard.
Click on Add on the next screen.
Click on all the visualizations you would like to add to the dashboard and X out of the window.
Save the dashboard.
Your dashboard is ready!
Observe as new events are generated and processed, the graphs in the dashboard get updated in real-time.
Please go to the Clean up section and clean up the resources in your account.