For this step you will need to Connect to your KafkaClientInstance via SSH in Cloud9
You can expand your Amazon MSK cluster using the CLI if you prefer to not uset he Console. Just a couple quick steps!
Get your clusters ARN - ARN referenced as $CLUSTER_ARN
Get the configuration hash for the cluster you want to modify, using the ARN from step 1
aws kafka describe-cluster --cluster-arn $CLUSTER_ARN --output json | jq ".ClusterInfo.CurrentVersion"
or simply
export CLUSTER_VERSION=$(aws kafka describe-cluster --cluster-arn $CLUSTER_ARN --output json | jq ".ClusterInfo.CurrentVersion" | tr -d \")
This will return a string similar to K138L1TOL8PIJT
Get the current number of brokers
aws kafka describe-cluster --cluster-arn $CLUSTER_ARN --output json | jq ".ClusterInfo.NumberOfBrokerNodes"
This will return the current number of brokers in the cluster
Set the new desired number of brokers. The number must be a multiple of your deployed subnets - ie: if you deploy in 3 subnets, you can only add in 3s. 2 subnets, add in 2s.
Remember to use the value from step 2 as the --current-version
if you didn’t set $CLUSTER_VERSION above
aws kafka update-broker-count --cluster-arn $CLUSTER_ARN --current-version $CLUSTER_VERSION --target-number-of-broker-nodes 9
After you run the command, it will return your cluster ARN, and a new value, a ClusterOperationARN. This is a value that is unique to this task (cluster expansion), and can be used to query for status.
ClusterOperationArn: arn:aws:kafka:us-east-1:xyz:cluster-operation/MSKCluster/0546f493-019f-475a-9903-272f0371ce19-6/9ab2c1a2-9a6c-456a-8ae1-1db4ddf4e357
Check to see how the expansion is progressing
Using the ClusterOperationARN, we can check and see the status of the change
aws kafka describe-cluster-operation --cluster-operation-arn arn:aws:kafka:us-east-1:xyz:cluster-operation/MSKCluster/0546f493-019f-475a-9903-272f0371ce19-6/9ab2c1a2-9a6c-456a-8ae1-1db4ddf4e357 --output json
If `OperationState` has the value of `UPDATE_IN_PROGRESS` wait a while, then run the command again. When it is complete, the status will be `UPDATE_COMPLETE` and the new brokers are ready for [partitions to be assigned](/en/addingbrokers/reassignpartitions.html).
Note
If you try and set the number of brokers to a number outside the bounds (explained above), you will see the following error:
An error occurred (BadRequestException) when calling the UpdateBrokerCount operation: The number of broker nodes must be a multiple of Availability Zones in the Client Subnets parameter.