Overview

Managing Client certificates for mutual authentication with Amazon MSK

It is a common security requirement to enable encryption-in-transit and authentication with Apache Kafka. Apache Kafka supports multiple authentication mechanisms including TLS mutual authentication using certificates, SASL (Simple Authorization Security Layer) PLAINTEXT, SASL Salted Challenge Response Authentication Mechanism (SCRAM), SASL Generic Security Services Application Program Interface (GSSAPI), SASL OAUTHBEARER (SASL mechanism for OAuth 2). Currently, Amazon Managed Streaming for Apache Kafka (Amazon MSK) supports encryption in transit with TLS and TLS mutual authentication with TLS certificates for client authentication.

Amazon MSK utilizes Amazon Certificate Manager Private Certificate Authority (ACM PCA) for TLS mutual authentication. For information about Private Certificate Authorities, see Creating and Managing a Private CA and see Certificate Authority for information on Certificate Authorities. The PCA can either be a root Certificate Authority (CA) or a subordinate Certificate Authority. If it is a root CA, you need to install a self-signed certificate (the console provides an easy mechanism to do that). If it is a subordinate CA, you can either choose an ACM PCA root or subordinate CA or an external CA (in this case, the external CA which can be your own CA will become the root of your certificate chain). In addition, for Amazon MSK to be able to use the ACM PCA, it needs to be in the same AWS account as the Amazon MSK cluster. However, the Apache Kafka clients, for example, the producers and consumers, schema registries, Kafka Connect or other Apache Kafka tools that need the end-entity certificates can be in an AWS account different from the AWS account that the ACM PCA is in. In that scenario, in order to be able to access the ACM PCA, they need to assume a role in the account the ACM PCA is in and has the required permissions as the ACM PCA does not support resource-based policies, only identity-based policies.

If encryption in-transit is enabled for an Amazon MSK cluster, Public TLS certificates from ACM are installed in the Amazon MSK Apache Kafka brokers keystores. If TLS mutual authentication is enabled for the Amazon MSK cluster, you need to provide an ACM PCA Amazon Resource Number (ARN) that the Amazon MSK cluster can utilize. The CA certificate and the certificate chain of the specified PCA are retrieved and installed in the truststores of the Amazon MSK Apache Kafka brokers.

On the clients, you need to generate a Private Key and create a CSR (Certificate Signing Request) that are used to get end-entity certificates issued by the ACM PCA specified for an Amazon MSK cluster. These certificates and their certificate chains are installed in the keystores on the client and are trusted by the Amazon MSK Apache Kafka brokers. The steps are documented in Client Authentication.