Setup Lambda

Steps we will perform in this section

  • Setup a Lambda function, a Kinesis Data Firehose delivery stream and a Lambda trigger or event source mapping between the Amazon MSK Apache Kafka topic and the Lambda function.

Use SAM (Serverless Application Model) to deploy resources

  • Go back to the AWS Cloud9 console. Click on MSKClient-Cloud9EC2Bastion.

  • Click on Open IDE (or go back to the browser tab if already open).

  • You should be connected to the KafkaClientEC2Instance instance. If not, ssh to the KafkaClientEC2Instance instance by pasting the ssh command you copied in the last section.

  • Go to the /tmp/kafka directory and clone the integration-sample-lambda-msk git repository.
    Note: Take a look at the code in GitHub . To see a summary of what the code is doing see Overview.

    cd /tmp/kafka
    git clone
    cd integration-sample-lambda-msk
  • Run the script. The script does the following. (You can cat the script to look at its contents.)

    1. It installs AWS SAM CLI if it is not installed.
    2. It creates an Amazon S3 bucket to be used for uploaded artifacts with a random prefix in its name.
    3. It uses sam to build the Lambda function and package its resources.
    4. It deploys the sam template and creates a CloudFormation stack with multiple resources. The resources include:
      • An IAM policy to be used by the Lambda function.
      • An IAM role with the policy to be used by the Lambda function.
      • An IAM role to be used by Amazon Kinesis Data Firehose.
      • A Kinesis Data Firehose delivery stream.
      • The Lambda function which will process records from a topic in Amazon MSK and send to Kinesis Data Firehose.
      • An EventSourceMapping mapping the Lambda function to the Amazon MSK Apache Kafka topic.

    Note: Change the values of MSK_STACK and KAFKA_CLIENT_STACK if you changed them from the default values.

    export MSK_STACK=MSK
    export VPC_STACK=$(aws cloudformation describe-stacks --stack-name $MSK_STACK --query 'Stacks[0].Outputs[?OutputKey==`VPCStackName`].OutputValue' --output text)
  • At this point, everything is setup and the Lambda function is already processing records from Amazon MSK. We will look at the environment in the next section.