While you did your homework and followed best practices, your workload has grown and you need to make your Amazon MSK cluster bigger to support additional producers, consumers, retention, etc.

This module will walk you through the process of adding brokers to your cluster, both on the Console and using the CLI

After you are done adding brokers, you will need to rebalance the cluster to assign partitions and replicas to the new brokers, otherwise they will just sit there empty.

  1. This is what the basic architecture for Amazon MSK looks like with 3 brokers. The red squares represent one of the 10 partitions of a test10 topic you will add, all distributed across the available brokers.:
    Brokers are running within the Amazon MSK VPC with ENIs connected in to your subnets. There is one broker running per configured availability zone.

  2. After we add another broker to each AZ in this exercise, it will look like this - note that the partitions haven’t moved and the new brokers have nothing assigned:
    There are new ENIs attached to your availabilty zone, and the new brokers are launched.

  3. Then we will run the partition reassignment, and it will move some of the partitions on to the new brokers so they are carrying traffic.

Important Tidbits

  • With Amazon MSK you can only scale in units that match your number of deployed AZs - if you deploy with 3 AZs, you can only scale up by 3 brokers at a time

  • When you add brokers, they will be idle until you assign partitions from the existing brokers to the new brokers, or add new topics. Each broker is assigned partitions, and producers write data to partitions. Thus if partitions are not mapped to brokers, brokers cannot accept, send or fetch requests from clients.

  • Migrating partitions will take time while data is replicated to the new brokers, and could impact the performance of the cluster

    • ideally don’t do this while under full load in production
    • You can rate limit replication to limit the risk of this
  • You are adding brokers to a cluster, so there will be an additional cost to operate your Amazon MSK cluster based on your instance size

A note and warning about Zookeeper

If you want to add brokers to the Amazon MSK cluster, you must use the MSK tools (Console, CLI) to do this - do not do it directly by building brokers and updating ZK. This will leave the cluster in an inconsistent state, and further operations on the cluster through MSK may remove the nodes you added manually.

As a general rule, don’t touch Zookeeper. There is no need to directly manage the Zookeeper cluster within your MSK deployment.