Kafka Connect

Starting Kafka Connect using TLS mutual authentication with an Amazon MSK Cluster

We will be starting Kafka Connect in distributed mode and utilizing TLS mutual authentication to connect to the Amazon MSK cluster.
Note: This lab does not deploy a connector on Kafka Connect. It merely shows how to configure and start Kafka Connect with TLS mutual authentication.

  • Go to the /tmp/kafka dir and edit the connect-distributed.properties file and edit the following properties.

    bootstrap.servers=<BootstrapBroker-String(TLS)>
    ssl.keystore.password=<password that you used for the keystore when running the authorization jar file>
    ssl.key.password=<password that you used for the keystore when running the authorization jar file>
    
  • Start the Kafka Connect service. It will use the ssl parameters from the /tmp/connect-distributed.properties file and connect to the Amazon MSK cluster using TLS mutual authentication.

    sudo systemctl start kafka-connect.service
    sudo systemctl status kafka-connect.service
    
    • This is the expected output from running these commands.


  • Make sure the Kafka Connect service started properly.

    cat /tmp/kafka/kafka-connect.log|grep Herder
    
    • You should see an output similar to this.
      Note: A Distributed herder coordinates with other workers to spread work across multiple processes. In the output you can see that the Herder started successfully and could connect to the Amazon MSK cluster using TLS mutual authentication and read the internal Kafka connect topics.