Amazon MSK Labs > Open monitoring Lab > Run Prometheus using Docker

Run Prometheus using Docker

Reuirements

Get your broker list and put it into your notepad

We will use Docker to simplify the installation of Prometheus. This is as simple as creating a configuration file, then running a docker command to start the service and you’re off to the races!

Process

Use Cloud9 to open a shell

You will be working in the Cloud9 Bash window for most of this lab - it’s a regular bash environment, but you may need to install tools manually if you want to use something that is missing.

This environment is different than the KafkaClientInstance, and will not have all the Kafka command line tools.
Create a prometheus directory to work in
```
 mkdir ~/prometheus
```
Create a service configuration file

Create prometheus server configuration - ~/prometheus/prometheus.yml

# file: prometheus.yml
# my global config
global:
  scrape_interval:     10s

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
  static_configs:
  # 9090 is the prometheus server port
  - targets: ['localhost:9090']
- job_name: 'broker'
  file_sd_configs:
  - files:
    - 'targets.json'

Create a file ~/prometheus/targets.json

Contents (replacing broker_dns_[1,2,N] with the values from the step above):

[
{
    "labels": {
    "job": "jmx"
    },
    "targets": [
    "broker_dns_1:11001",
    "broker_dns_2:11001",
    .
    "broker_dns_N:11001"
    ]
},
{
    "labels": {
    "job": "node"
    },
    "targets": [
    "broker_dns_1:11002",
    "broker_dns_2:11002",
    .
    "broker_dns_N:11002"
    ]
}
]

Example of a completed file:

[
{
    "labels": {
    "job": "jmx"
    },
    "targets": [
        "b-3.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001",
        "b-2.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001",
        "b-1.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11001"
    ]
},
{
    "labels": {
    "job": "node"
    },
    "targets": [
        "b-3.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002",
        "b-2.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002",
        "b-1.mskworkshopclustercliv.xyz.c6.kafka.us-east-1.amazonaws.com:11002"
    ]
}
]

Run prometheus

We are going to fire up Prometheus running docker. This will pull down the container and run it, mounting the config files created above into the container, and exposing the service on port 9090

sudo docker run -d -p 9090:9090 --name=prometheus -v /home/ec2-user/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v /home/ec2-user/prometheus/targets.json:/etc/prometheus/targets.json prom/prometheus --config.file=/etc/prometheus/prometheus.yml

The command will return your containerID, eg:

db5fa73d5a197935cd7294b1db5b3a4d9057afe0ff2624514d28787fb3f778e6

Note: this is an example - use the output from the docker run command

Some Docker Tips before going on

If you haven’t used docker before, or need a refresh, some basic operational processes:

The container will run in the background. You can view the logs by running

sudo docker logs <containerID>

eg:

sudo docker logs db5fa73d5a197935cd7294b1db5b3a4d9057afe0ff2624514d28787fb3f778e6

You can add the command line option -f to follow the logs

If you want to get a shell in the container to look at configs, check processes, etc, you can do:

sudo docker exec -it <containerID> /bin/sh

If you want to stop the container for some reason

sudo docker stop prometheus

If you want to start the container

sudo docker start prometheus

If you want to get rid of the container entirely you need to stop it, then remove the container

sudo docker rm prometheus

Update the security group on your KafkaClientInstance
- In the Console, open the EC2 service, select Security Groups
- Click the checkbox next to aws-cloud9-msklab... and then click Actions and Edit Inbound Rules
- Click Add Rule and add the following rules:
- Rule 1 - Prometheus
  - Type: Custom TCP
  - Port Range: 9090
  - Source: MyIP and add the security group aws-cloud9-msklab... (the group you’re editing) as well
  - Description: Prometheus
- Rule 2 - Grafana
  - Type: Custom TCP
  - Port Range: 3000
  - Source: MyIP
  - Description: Grafana
- Click Save Rules in the bottom right of the window

Connect to Prometheus

In the Console, switch to the EC2 service, click on your aws-cloud9-msklab... instance, then copy the IPv4 Public IP address in the pane below

Flip to your web browser, browse to the that Public IPv4 address on port 9090

Example: http://1.2.3.4:9090

You will now be in the Prometheus web interface

Review incoming metrics

You can do some basic graphing and metrics collection in the Prometheus webUI. But we will use Grafana for more advanced dashboarding

In the Graph pane in Prometheus (selected from the menu along the top), in the Expression bar, start typing kafka. This will allow you to browse all the metrics being pull in to Prometheus

Enter kafka_controller_KafkaController_Value{name="OfflinePartitionsCount"} to get a graph of offline partitions

Click ‘Graph’ tab under the Expression bar and review a simple graph of the metric

That’s it

The Prometheus web interface is pretty simple, and useful really for validation. To do anything with the data you’ll need to connect a tool that can do dashboarding, such as Grafana. Checkout our next exercise for that!

Resources

Install Prometheus