Overview

This module will walk you through how to use both the Console and AWS CLI to create a custom configuration and an Amazon MSK Cluster.

Custom Configuration

The custom configuration will enable us to provide a special configuration to the cluster. Review the available options to make sure you have what you need.

Cluster

The cluster will be deployed into an existing VPC, with brokers deployed in 3 private subnets (one per AZ). We will use m5.large nodes for this exercise. If you are using an existing VPC, please ensure that there is a private subnet in each AZ into which you can deploy.

When doing the CLI deploy, you will need to provide a number of inputs. It’s handy to have open a text editor of your choice to keep track of the details.

Architecture

msk-arch

The diagram demonstrates the interaction between the following components:

  • Broker nodes - When creating an Amazon MSK cluster, you specify how many broker nodes you want Amazon MSK to create in each Availability Zone. In the example cluster shown in this diagram, there's one broker per Availability Zone. Each Availability Zone has its own virtual private cloud (VPC) subnet. Amazon MSK offers a 3 AZ or a 2 AZ Apacke Kafka node layout. Currently, a 1 AZ (single AZ) Amazon MSK cluster is not available.

  • ZooKeeper nodes - Amazon MSK also creates the Apache ZooKeeper nodes for you. Apache ZooKeeper is an open-source server that enables highly reliable, distributed coordination. There is a dedicated ZooKeeper ensemble created for each Amazon MSK cluster that is fully obfuscated and included with each cluster at no additional cost.

  • Producers, consumers, and topic creators - Amazon MSK lets you use Apache Kafka data-plane operations to create topics and to produce and consume data.

  • AWS CLI - You can use the AWS Command Line Interface (AWS CLI) or the APIs in the SDK to perform control-plane operations. For example, you can use the AWS CLI or the SDK to create or delete an Amazon MSK cluster, list all the clusters in an account, or view the properties of a cluster.