Install Cruise Control


On the cloud9 instance open a terminal window:

  1. Install Java8 on the cloud9 instance
sudo yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
sudo alternatives --config java 

You will be asked to select a version. It should be #2, /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java.


mitchadmin:/usr/lib/jvm $ sudo alternatives --config java                                                                                                                                       

There are 2 programs which provide 'java'.

  Selection    Command
*+ 1           /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
   2           /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java

Enter to keep the current selection[+], or type selection number: 2
mitchadmin:/usr/lib/jvm $ java -version
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

Repeat the process for javac.

mitchadmin:~/cruise-control-2.4.25 (master) $ sudo update-alternatives --config javac                                                                                                           

There are 2 programs which provide 'javac'.

  Selection    Command
*+ 1           /usr/lib/jvm/java-1.7.0-openjdk.x86_64/bin/javac
   2           /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac

Enter to keep the current selection[+], or type selection number: 2
  1. Download a cruise-control release from

    cd ~/
  2. You will need to extract the source code from the file you just downloaded and go into its directory

tar -xvzf 2.4.25.tar.gz
cd cruise-control-2.4.25/
  1. To build Cruise Control you must initialize it as a git repo,
git init && git add . && git commit -m "Init local repo." && git tag -a 2.0.130 -m "Init local version."
  1. Build cruise control with
./gradlew jar copyDependantLibs
  1. Get the Amazon MSK ClusterArn. Note that if you have more then one active Amazon MSK cluster you will need to set MSKARN manually.
export MSKARN=`aws kafka list-clusters|grep ClusterArn|cut -d ':' -f 2-|cut -d ',' -f 1 | sed -e 's/\"//g'`
  1. Set the bootstrap.servers to be used for configuring Cruise Control.
export MYBROKERS=`aws kafka get-bootstrap-brokers --cluster-arn $MSKARN|grep BootstrapBrokerString|grep 9092| cut -d ':' -f 2- | sed -e 's/\"//g' | sed -e 's/,$//'`
  1. Set the Zookeeper Connect string for configuring Cruise Control.
export ZOOKEEPER=`aws kafka describe-cluster --cluster-arn $MSKARN|grep ZookeeperConnectString|grep -v Tls|cut -d ':' -f 2-|sed 's/,$//g'|sed -e 's/\"//g'`

  1. Edit config/ to fit your environment.
sed -i "s/localhost:9092/${MYBROKERS}/g" config/
sed -i "s/localhost:2181/${ZOOKEEPER}/g" config/
sed -i "s/com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler/com.linkedin.kafka.cruisecontrol.monitor.sampling.prometheus.PrometheusMetricSampler/g" config/
sed -i "s/webserver.http.port=9090/webserver.http.port=9091/g" config/ 
sed -i "s/capacity.config.file=config\/capacityJBOD.json/capacity.config.file=config\/capacityCores.json/g" config/
echo "prometheus.server.endpoint=localhost:9090" >> config/
mkdir logs; touch logs/kafka-cruise-control.out

Note: that prometheus.server.endpoint= is the host that is currently running prometheus as built in the Open Monitoring Lab.

  1. Modify config/capacityCores.json. For m5.large instance type:
      "brokerId": "-1",
      "capacity": {
        "DISK": "10737412742445",
        "CPU": {"num.cores": "2"},
        "NW_IN": "1073741824",
        "NW_OUT": "1073741824"
      "doc": "This is the default capacity. Capacity unit used for disk is in MB, cpu is in number of cores, network throughput is in KB."
  1. Run Cruise Control from inside the decompressed Cruise Control directory.
./ -daemon config/
  1. After a few minutes, in tail logs/kafkacruisecontrol.log you should see something like:
[2020-11-09 16:01:30,060] INFO Kicking off metric sampling for time range [1604937570060, 1604937690060], duration 120000 ms with timeout 120000 ms. (com.linkedin.kafka.cruisecontrol.monitor.sampling.MetricFetcherManager)
[2020-11-09 16:01:30,141] INFO Generated 279 partition metric samples and 6 broker metric samples for timestamp 1604937690000. (com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsProcessor)
[2020-11-09 16:01:30,142] INFO Collected 279 partition metric samples for 279 partitions. Total partition assigned: 279. (com.linkedin.kafka.cruisecontrol.monitor.sampling.SamplingFetcher)

This verifies that Cruise Control is collecting metrics from Prometheus.