Self-managed Kafka cluster

Using LocalStack lambda with self-managed Kafka cluster

Tags:

2 minute read

LocalStack does not currently support AWS MSK out of the box, but you can run your own self-managed Kafka cluster and integrate it with your own applications.

Running self-managed Kafka

You can find the example Docker Compose file which contains a single-noded ZooKeeper and a Kafka cluster and a simple LocalStack setup as well as Kowl, an Apache Kafka Web UI.

Run Docker Compose:

$ docker-compose up -d

Create the Lambda function:

$ awslocal lambda create-function \
    --function-name fun1 \
    --handler lambda.handler \
    --runtime python3.8 \
    --role r1 \
    --zip-file fileb://lambda.zip
{
    "FunctionName": "fun1",
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1",
    "Runtime": "python3.8",
    "Role": "r1",
    "Handler": "lambda.handler",
    "CodeSize": 294,
    "Description": "",
    "Timeout": 3,
    "LastModified": "2021-05-19T02:01:06.617+0000",
    "CodeSha256": "/GPsiNXaq4tBA4QpxPCwgpeVfP7j+1tTH6zdkJ3jiU4=",
    "Version": "$LATEST",
    "VpcConfig": {},
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "d85469d2-8558-4d75-bc0e-5926f373e12c",
    "State": "Active",
    "LastUpdateStatus": "Successful",
    "PackageType": "Zip"
}

Create an example secret:

$ awslocal secretsmanager create-secret --name localstack
{
    "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI",
    "Name": "localstack",
    "VersionId": "32bbb8e2-46ee-4322-b3d5-b6459d54513b"
}

Create an example Kafka topic:

$ docker exec -ti kafka kafka-topics --zookeeper zookeeper:2181 --create --replication-factor 1 --partitions 1 --topic t1
Created topic t1.

Create the event source mapping to your local kafka cluster:

$ awslocal lambda create-event-source-mapping \
    --topics t1 \
    --source-access-configuration Type=SASL_SCRAM_512_AUTH,URI=arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI \
    --function-name arn:aws:lambda:us-east-1:000000000000:function:fun1 \
    --self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:9092"]}}'
{
    "UUID": "4a2b0ea6-960c-4847-8684-465876dd6dbd",
    "BatchSize": 100,
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1",
    "LastModified": "2021-05-19T04:02:49+02:00",
    "LastProcessingResult": "OK",
    "State": "Enabled",
    "StateTransitionReason": "User action",
    "Topics": [
        "t1"
    ],
    "SourceAccessConfigurations": [
        {
            "Type": "SASL_SCRAM_512_AUTH",
            "URI": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI"
        }
    ],
    "SelfManagedEventSource": {
        "Endpoints": {
            "KAFKA_BOOTSTRAP_SERVERS": [
                "localhost:9092"
            ]
        }
    }
}

Additionally visit http://localhost:8080 for Kowl’s UI.

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified December 1, 2022: LocalStack Beta Docs (#337) (28576f89)