Access AWS MSK Kafka cluster with IAM authentication using Teleport VNET

[milos-teleport]

Goal

Access AWS MSK Kafka with VNET while leveraging AWS IAM Authentication for AWS MSK

Prerequisites

• MSK Kafka cluster stood up and running. This guide uses Kafka version 3.6.0 on the cluster.
• Existing Teleport Enterprise self-hosted cluster or Cloud instance - v16 or greater
• EC2 Instance available with network connectivity to the MSK Kafka cluster. This EC2 instance will be used to provision a Teleport agent with app access.
• Teleport Connect installed and working at the workstation - v16 or greater

Notes

At the time of writing this, November 2024, VNET is only supported on macOS with support for Linux and Windows planned.

Overview

We will use Teleport's VNET feature to access an AWS MSK Kafka cluster.

This guide will cover the following:

1. Setting up VNET on our Teleport cluster
2. Setting up the EC2 Teleport agent for app access to MSK
3. Setting up the workstation with appropriate AWS permissions and Kafka utilities
4. Connect to VNET and then to our Kafka cluster using the broker endpoints provided by AWS

Prepare VNET

kind: vnet_config
version: v1
metadata:
  name: kafka-vnet-test
spec:
  custom_dns_zones:
  # Replace with your kafka domain of your MSK Kafka cluster
  - suffix: examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com

Apply the above config

tctl create vnet-config.yaml

Prepare the EC2 Teleport agent instance

This EC2 instance must reside in a subnet which can reach the MSK Kafka endpoints

Create the token

tctl tokens add --type node,app --value ec2-join

After launching an EC2 Teleport agent instance, run this

export TELEPORT_DOMAIN=myteleport.teleport.sh
export TELEPORT_VERSION="$(curl https://$TELEPORT_DOMAIN/v1/webapi/automaticupgrades/channel/default/version | sed 's/v//')"
curl https://cdn.teleport.dev/install-v${TELEPORT_VERSION}.sh | bash -s ${TELEPORT_VERSION} enterprise
sudo teleport configure --proxy myteleport.teleport.sh:443 \
  --token ec2-join \
  --roles node,app \
  --output=file:///etc/teleport.yaml
sudo systemctl enable --now teleport
sudo journalctl -fu teleport

We need to add all our AWS MSK Kafka brokers as "apps" definitions in Teleport.

To do this, add the following block to /etc/teleport.yaml. This was not doable with CLI because public_addr is not yet a parameter in teleport configure

app_service:
  enabled: "yes"
  apps:
  - name: "b-1"
    uri: tcp://b-1.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098
    public_addr: b-1.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com
  - name: "b-2"
    uri: tcp://b-2.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098
    public_addr: b-2.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com
  - name: "b-3"
    uri: tcp://b-3.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098
    public_addr: b-3.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com

Workstation

Your workstation AWS credentials need to have IAM access for AWS MSK. Refer to the permission documentation here

Reference: Video from AWS describing generic Kafka access using AWS IAM authentication

Install java-17 which is supported by Kafka 3.6.1

RHEL-compatibles:

sudo dnf install java-17

...or on macOS

brew install openjdk@17

# This instruction can be found in `brew info openjdk@17`
# I think alternatively env vars can be set. This is system-wide
sudo ln -sfn /opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-17.jdk

Grab Kafka 3.6.1 using this command:

# I did this in the homedir of ec2-user
wget https://archive.apache.org/dist/kafka/3.6.1/kafka_2.13-3.6.1.tgz
tar -xzf kafka_2.13-3.6.1.tgz

Get the aws-msk-iam-auth jar file from here. For example:

wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.2.0/aws-msk-iam-auth-2.2.0-all.jar
# And then move the file to libs
mv aws-msk-iam-auth-2.2.0-all.jar kafka_2.13-3.6.1/libs/

config.txt

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Connect to VNET

• Open Teleport Connect
• Log-out and log back in - if you have made recent role/configuration changes
• Click the VNET icon. It should show the AWS MSK domain we configured early in the guide.

Commands

# Run these in all the shell tabs/instances you wish to run the kafka commands
export TELEPORT_DOMAIN="myteleport.teleport.sh"
# Copy from AWS console or AWS CLI:
# aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:eu-west-1:278576220453:cluster/examplecluster/00fa0602-d7ab-4170-b57f-87304ed2cc3c-8
export BOOTSTRAP_SERVERS="b-3.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098,b-1.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098,b-2.examplecluster.xxxxxx.yy.kafka.eu-west-1.amazonaws.com:9098"
# Adjust this export DIR if needed
export KAFKA_BIN_DIR=$HOME/kafka_2.13-3.6.1/bin/

Create a topic

$KAFKA_BIN_DIR/kafka-topics.sh \
        --bootstrap-server "$BOOTSTRAP_SERVERS"\
        --create \
        --replication-factor 3 \
        --partitions 1 \
        --topic exampletopic \
        --command-config config.txt

List topics

$KAFKA_BIN_DIR/kafka-topics.sh \
        --bootstrap-server "$BOOTSTRAP_SERVERS" --list \
        --command-config config.txt

Start a consumer in a separate shell (remember to run the export commands)

# Consumer
$KAFKA_BIN_DIR/kafka-console-consumer.sh \
        --bootstrap-server "$BOOTSTRAP_SERVERS" \
        --topic exampletopic \
        --from-beginning \
        --consumer.config config.txt

Start a producer, and start typing and hitting ENTER to add messages to the Kafka topic.

#Producer
$KAFKA_BIN_DIR/kafka-console-producer.sh \
        --bootstrap-server "$BOOTSTRAP_SERVERS" \
        --topic exampletopic \
        --producer.config config.txt

Check back on the consumer. You should see the messages appear there.