Your Amazon MSK cluster has grown, so you’ve added new brokers. But, new brokers will just sit there idle until you assign them some work. This lab will show you how to accomplish this.
You will need a text editor to keep notes. For this section you will need:
$MYZK
in this pageEach broker is assigned a number of partitions - a partition is assigned to a broker at the creation time of the topic (which includes partition creation) and/or addition of partitions to a cluster.
We are going to use a native kafka tool to do this assignment. The basic steps are:
Generate a list of topics that you want to move
Use kafka-reassign-partitions
tool to generate a reassignment template
If the reassignment plan looks good, we will execuate the template and start the reassignment
Finally, we can monitor the status of the reassignment as we wait for it to complete
We need to build a list of topics we want to move, and get an understanding of where things are now so we can compare when we are done. Lets use the topic you created in the Overview section (should be called test10):
Get the current partition allocation
bin/kafka-topics.sh --zookeeper $MYZK --describe --topic test10
Notice that the leaders are all brokers with the IDs of 1, 2, or 3.
Now we will decide what topics we want to migrate, then create a file to describe the topics to move - call this topics-to-move.json. For this example, we will use the topic test10 from above. You could also use a topic like __consumer_offsets if you want more than one to migrate.
So for test10, your topics-to-move.json
file would look like:
{ "topics": [ { "topic" : "test10"}], "version":1}
We know how things look, we know what we want to move, now we need to generate a candidate assignment, or a new proposed layout for the partitions. We will use a tool that comes with Kafka - kafka-reassign-partitions.sh
in –generate mode, along with our Zookeeper connection string and our topics-to-move.json
file. You will also need to pass in the list of brokers you want to spead the partitions across - ie: “1,2,3,4,5,6” if you’ve expanded your cluster to 6 brokers
bin/kafka-reassign-partitions.sh --zookeeper $MYZK --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5,6" --generate
The output will be a big long and kinda ugly pair on JSON objects - what you have now, and what the tool is proposing for the end state. Having the current state is handy if you need to roll back, so copy and paste that somewhere for safekeeping. As well, copy and paste the new assignments to a file - expand-cluster-reassignment.json
Tip: If you want to see your file in a more readable way, use jq:
cat expand-cluster-reassignment.json | jq "."
This will give you nicely formatted text.
That’s the end of the prep step - now we’re on to implementation
Time to pull the trigger and make the change on the cluster!
For this you will use kafka-reassign-partitions.sh
again, this we input the expand-cluster-reassignment.json
file, and tell it to --execute
bin/kafka-reassign-partitions.sh --zookeeper $MYZK --reassignment-json-file expand-cluster-reassignment.json --execute
Once you’ve run this, and if everything goes as planned, it will return Successfully started reassignment of partitions.
We can reuse the command above, changing --execute
to --verify
to see how the assignements are going
bin/kafka-reassign-partitions.sh --zookeeper $MYZK --reassignment-json-file expand-cluster-reassignment.json --verify
Reassignment of partition test10-7 completed successfully
Reassignment of partition test10-2 completed successfully
Reassignment of partition test10-8 completed successfully
Reassignment of partition test10-3 completed successfully
Reassignment of partition test10-9 completed successfully
Reassignment of partition test10-0 completed successfully
Reassignment of partition test10-4 completed successfully
Reassignment of partition test10-6 completed successfully
Reassignment of partition test10-1 completed successfully
Reassignment of partition test10-5 completed successfully
You are done! That’s it. You can see the state with kafka-topics.sh
:
bin/kafka-topics.sh --zookeeper $MYZK --describe --topic test10
Notice that now the leaders are all 6 brokers, not just the three we saw in the initial part of this lab
If for some reason you wanted to roll back to the previous configuration, you stashed the original state in a file in step 3 above. Also, the output of the --execute
had the original state in it. We can use that data to reassign the partitions back. Put the contents in to the file expand-cluster-reassignment.json
and then rerun your --execute
command:
bin/kafka-reassign-partitions.sh --zookeeper $MYZK --reassignment-json-file expand-cluster-reassignment.json --execute
Example:
Note that this only rolls back the partition assignment - it does not roll back the broker creation/addition to the cluster.