We will use Docker to simplify the installation of Prometheus. This is as simple as creating a configuration file, then running a docker
command to start the service and you’re off to the races!
Use Cloud9 to open a shell
You will be working in the Cloud9 Bash window for most of this lab - it’s a regular bash environment, but you may need to install tools manually if you want to use something that is missing.
This environment is different than the KafkaClientInstance, and will not have all the Kafka command line tools.
Create a prometheus directory to work in
mkdir ~/prometheus
Create a service configuration file
Create prometheus server configuration - ~/prometheus/prometheus.yml
# file: prometheus.yml
# my global config
global:
scrape_interval: 10s
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
# 9090 is the prometheus server port
- targets: ['localhost:9090']
- job_name: 'broker'
file_sd_configs:
- files:
- 'targets.json'
~/prometheus/targets.json
Contents (replacing broker_dns_[1,2,N]
with the values from the step above):
[
{
"labels": {
"job": "jmx"
},
"targets": [
"broker_dns_1:11001",
"broker_dns_2:11001",
.
"broker_dns_N:11001"
]
},
{
"labels": {
"job": "node"
},
"targets": [
"broker_dns_1:11002",
"broker_dns_2:11002",
.
"broker_dns_N:11002"
]
}
]
Example of a completed file:
[
{
"labels": {
"job": "jmx"
},
"targets": [
"b-3.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001",
"b-2.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001",
"b-1.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001"
]
},
{
"labels": {
"job": "node"
},
"targets": [
"b-3.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002",
"b-2.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002",
"b-1.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002"
]
}
]
We are going to fire up Prometheus running docker. This will pull down the container and run it, mounting the config files created above into the container, and exposing the service on port 9090
sudo docker run -d -p 9090:9090 --name=prometheus -v /home/ec2-user/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /home/ec2-user/prometheus/targets.json:/etc/prometheus/targets.json prom/prometheus --config.file=/etc/prometheus/prometheus.yml
The command will return your containerID, eg:
db5fa73d5a197935cd7294b1db5b3a4d9057afe0ff2624514d28787fb3f778e6
Note: this is an example - use the output from the docker run
command
If you haven’t used docker before, or need a refresh, some basic operational processes:
The container will run in the background. You can view the logs by running
sudo docker logs <containerID>
eg:
sudo docker logs db5fa73d5a197935cd7294b1db5b3a4d9057afe0ff2624514d28787fb3f778e6
You can add the command line option -f
to follow the logs
If you want to get a shell in the container to look at configs, check processes, etc, you can do:
sudo docker exec -it <containerID> /bin/sh
If you want to stop the container for some reason
sudo docker stop prometheus
If you want to start the container
sudo docker start prometheus
If you want to get rid of the container entirely you need to stop it, then remove the container
sudo docker rm prometheus
Update the security group on your KafkaClientInstance
In the Console, open the EC2 service, select Security Groups
Click the checkbox next to aws-cloud9-msklab...
and then click Actions
and Edit Inbound Rules
Click Add Rule and add the following rules:
Rule 1 - Prometheus
aws-cloud9-msklab...
(the group you’re editing) as wellRule 2 - Grafana
Click Save Rules
in the bottom right of the window
aws-cloud9-msklab...
instance, then copy the IPv4 Public IP address in the pane belowExample: http://1.2.3.4:9090
You will now be in the Prometheus web interface
You can do some basic graphing and metrics collection in the Prometheus webUI. But we will use Grafana for more advanced dashboarding
Graph
pane in Prometheus (selected from the menu along the top), in the Expression
bar, start typing kafka
. This will allow you to browse all the metrics being pull in to PrometheusEnter kafka_controller_KafkaController_Value{name="OfflinePartitionsCount"}
to get a graph of offline partitions
Click ‘Graph’ tab under the Expression
bar and review a simple graph of the metric
The Prometheus web interface is pretty simple, and useful really for validation. To do anything with the data you’ll need to connect a tool that can do dashboarding, such as Grafana. Checkout our next exercise for that!