Expand storage using the CLI

As a prerequisite for this, you will need to have:

You will also need to know what size you want the disks to be - you will provide the new value, not an increment to grow by. You can get the current value with:

    $ aws kafka describe-cluster --cluster-arn $CLUSTER_ARN --output json | grep VolumeSize
                "VolumeSize": 2000

This value will be <New Volume size> below


  1. Get your Current-cluster-version, which is a hash that identifies your config

     $ aws kafka describe-cluster <ClusterARN>  --output json | grep CurrentVersion
     "CurrentVersion": "K3AEGXETSR30VB",
    

or simply

export CLUSTER_VERSION=$(aws kafka describe-cluster --cluster-arn $CLLUSTER_ARN --output json | jq ".ClusterInfo.CurrentVersion" | tr -d \")
  1. Expand the disk volume size

Remember to change <New volume size> to the size you want

    $ aws kafka update-broker-storage --cluster-arn $CLUSTER_ARN --current-version $CLUSTER_VERSION --target-broker-ebs-volume-info '{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <New volume size >}' 

This will return a JSON object that contains an ARN for the action that’s taking place. Copy this for the next step, it will be the <cluster operations ARN>

  1. Monitor the progress of the change

     $ aws kafka describe-cluster-operation --cluster-operation-arn "<cluster operations ARN>"
    

This will give you an output similar to this:

$ aws kafka describe-cluster-operation --cluster-operation-arn $CLUSTER_ARN
{
    "ClusterOperationInfo": {
        "ClientRequestId": "390aaa23-579a-49b7-bb33-e5f39f68ecd1",
        "ClusterArn": "arn:aws:kafka:us-east-1:1553xyz16955788:cluster/ToddTest2/c2209be6-9efd-4cd8-9602-60e6220fd84c-6",
        "CreationTime": "2020-02-21T20:31:52.872Z",
        "OperationArn": "arn:aws:kafka:us-east-1:xyz:cluster-operation/ToddTest2/c2209be6-9efd-4cd8-9602-60e6220fd84c-6/8ce20ff9-6617-4139-a49f-e19c9175d4b4",
        "OperationState": "UPDATE_IN_PROGRESS",
        "OperationType": "UPDATE_BROKER_EBS_VOLUME",
        "SourceClusterInfo": {
            "BrokerEBSVolumeInfo": [
                {
                    "KafkaBrokerNodeId": "All",
                    "VolumeSizeGB": 2000
                }
            ]
        },
        "TargetClusterInfo": {
            "BrokerEBSVolumeInfo": [
                {
                    "KafkaBrokerNodeId": "All",
                    "VolumeSizeGB": 2500
                }
            ]
        }
    }   


You can see the config you've pushed to the cluster to be implemented, as well as the `OperationState` - it will say `UPDATE_IN_PROGRESS` until it's done, when it will change to "UPDATE_COMPLETE"

Resources