Setup Secrets

Create KMS (Key Management Service) CMK (Customer Master Key)

  • Go back to the AWS Cloud9 console. Click on MSKClient-Cloud9EC2Bastion.

  • Click on Open IDE (or go back to the browser tab if already open).


  • Go to the terminal.

  • Run the following commands.

    export stack_name=MSKClient
    export logical_resource_id=EC2Role
    export accountid=$(aws2 sts get-caller-identity --query Account --output text)
    export ec2role=$(aws2 cloudformation describe-stack-resource --stack-name $stack_name --logical-resource-id $logical_resource_id --query StackResourceDetail.PhysicalResourceId --output text)
    
    key_policy_param='{"Id":"key-cmk-saslscram-policy","Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::%s:root"},"Action":"kms:*","Resource":"*"},{"Effect":"Allow","Principal":{"AWS":  ["arn:aws:iam::%s:role/Admin","arn:aws:iam::%s:role/%s"]},"Action":["kms:Create*","kms:Describe*","kms:Enable*","kms:List*","kms:Put*","kms:Update*","kms:Revoke*","kms:Disable*","kms:Get*","kms:Delete*","kms:TagResource",   "kms:UntagResource","kms:ScheduleKeyDeletion","kms:CancelKeyDeletion"],"Resource":"*"},{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::%s:role/aws-service-role/kafka.amazonaws.com/AWSServiceRoleForKafka","arn:aws:iam::%s:role/Admin"]},"Action":["kms:Encrypt","kms:Decrypt","kms:ReEncrypt*","kms:GenerateDataKey*","kms:DescribeKey"],"Resource":"*"},{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::%s:role/aws-service-role/kafka.amazonaws.com/AWSServiceRoleForKafka","arn:aws:iam::%s:role/Admin"]},"Action":["kms:CreateGrant","kms:ListGrants","kms:RevokeGrant"],"Resource":"*","Condition":{"Bool":{"kms:GrantIsForAWSResource":"true"}}}]}'
    
    key_policy=$(printf "$key_policy_param" "$accountid" "$accountid" "$accountid" "$ec2role" "$accountid" "$accountid" "$accountid" "$accountid")
    
    key=$(aws2 kms create-key --policy "$key_policy" --description "SASLSCRAM Key for Amazon MSK")
    sasl_scram_key_id=$(echo $key|jq .KeyMetadata.KeyId --raw-output)
    
    aws2 kms create-alias --alias-name alias/saslscrammsk --target-key-id $sasl_scram_key_id
    
  • Create a grant on the key for the EC2 role for the KafkaClientEC2Instance.

    aws kms create-grant --key-id $sasl_scram_key_id --grantee-principal arn:aws:iam::$accountid:role/$ec2role --operations Decrypt GenerateDataKey
    

Create secrets for Amazon MSK SASL/SCRAM users in AWS Secrets Manager

  • We will create two SASL/SCRAM users, nancy and alice. alice is the administrator and hence will get all cluster level permissions on the Amazon MSK cluster but no topic read or write permissions while nancy will just have permissions to read and write from a specific topic called test. In addition, we will add different resource policies to the secrets so that the secret for Nancy is readable by the EC2 Role associated with the KafkaClientEC2Instance and the secret for Alice is not readable by the EC2 Role associated with the KafkaClientEC2Instance.

  • Run the following commands to create a secret for user nancy

    export secret_name=AmazonMSK_nancy
    export secret_string='{"username":"nancy","password":"nancy-secret"}'
    
    result=$(aws2 secretsmanager create-secret --name $secret_name --description "Amazon MSK sasl/scram user nancy" --kms-key-id $sasl_scram_key_id --secret-string $secret_string)
    export secret_arn=$(echo $result|jq .ARN --raw-output)
    
    resource_policy_param='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"kafka.amazonaws.com","AWS":"arn:aws:iam::%s:role\/%s"},"Action":"secretsmanager:getSecretValue","Resource":"%s"}]}'
    
    export resource_policy=$(printf "$resource_policy_param" "$accountid" "$ec2role" "$secret_arn")
    
    aws2 secretsmanager put-resource-policy --secret-id $secret_name --resource-policy $resource_policy
    
  • Run the following commands to create a secret for user alice

    export secret_name=AmazonMSK_alice
    export secret_string='{"username":"alice","password":"alice-secret"}'
    
    result=$(aws2 secretsmanager create-secret --name $secret_name --description "Amazon MSK sasl/scram user alice" --kms-key-id $sasl_scram_key_id --secret-string $secret_string)
    export secret_arn=$(echo $result|jq .ARN --raw-output)
    
    resource_policy_param='{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"kafka.amazonaws.com"},"Action":"secretsmanager:getSecretValue","Resource":"%s"}]}'
    
    export resource_policy=$(printf "$resource_policy_param" "$secret_arn")
    
    aws2 secretsmanager put-resource-policy --secret-id $secret_name --resource-policy $resource_policy
    
  • Try retrieving the secret value for nancy.

    aws2 secretsmanager get-secret-value --secret-id AmazonMSK_nancy
    
    • You should be able to successfully retrieve it and see something like the following.


  • Try retrieving the secret value for alice.

    aws2 secretsmanager get-secret-value --secret-id AmazonMSK_alice
    
    • You should get an authorization error and see something like the following.


    • This illustrates how you can control who has access to the secret at a granular secret by secret level using IAM Resource based policies.

Associate secrets with Amazon MSK

  • The secrets need to be associated with the Amazon MSK cluster to be available for SASL/SCRAM authentication.

  • Run the following commands.

    export stack_name=MSK
    export logical_resource_id=MSKCluster
    export msk_cluster_arn=$(aws2 cloudformation describe-stack-resource --stack-name $stack_name --logical-resource-id $logical_resource_id --query StackResourceDetail.PhysicalResourceId --output text)
    export secret_arn_list=$(aws2 secretsmanager list-secrets --query 'SecretList[?starts_with(Name, `AmazonMSK_`) == `true`].ARN' --output text)
    aws2 kafka batch-associate-scram-secret --cluster-arn $msk_cluster_arn --secret-arn-list $secret_arn_list