/
kafka Setup and App Cheat Sheet

kafka Setup and App Cheat Sheet

Owned by Selvyn Wright
Jul 09, 2024
3 min read

Something on Kafka

Installing kafka and running kafka

docker pull bashj79/kafka-kraft docker run -p 9092:9092 -d --name myKafka bashj79/kafka-kraft

Create a topic

docker exec -it myKafka /bin/bash cd /opt/kafka/bin

create topic 'my-first-topic'

sh kafka-topics.sh --bootstrap-server localhost:9092 --create --topic my-test-topic --partitions 1 --replication-factor 1

list topics

sh kafka-topics.sh --bootstrap-server localhost:9092 --list

send messages to the topic

sh kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-test-topic Hello World Here we are in DDAT

Using the Offset Explorer

Courtesy of Ralf Ueberfuhr

The Offset Explorer (formerly: Kafka Tool) is a GUI application for managing Kafka. We can download and install it quickly. Then, we create a connection and specify the host and port of the Kafka broker

image-20240704-085158.png

You will see

image-20240708-171119.png

Yu can use this to administer the channels (kafka calls them logs)

Using UI for Apache Kafka (Kafka UI) - my preferred option

Courtesy of Ralf Ueberfuhr

The UI for Apache Kafka (Kafka UI) is a web UI, implemented with Spring Boot and React, and provided as a Docker container for a simple installation with the following command:

docker run -it -p 8080:8080 --name kafka-ui -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui

We can then open the UI in the browser using http://localhost:8080 and define a cluster, as this picture shows:

image-20240704-085530.png

Because the Kafka broker runs in a different container than the Kafka UI’s backend, it will not have access to localhost:9092. We could instead address the host system using host.docker.internal:9092, but this is just the bootstrapping URL.

Unfortunately, Kafka itself will return a response that leads to a redirection to localhost:9092 again, which won’t work. If we do not want to configure Kafka (because this would break with the other clients then), we need to create a port forwarding from the Kafka UI’s container port 9092 to the host systems port 9092. The following sketch illustrates the connections (Courtesy of Ralf Ueberfuhr):

image-20240704-085620.png

We can set up this container-internal port forwarding, e.g. using socat. We have to install it within the container (Alpine Linux), so we need to connect to the container’s bash with root permissions. So we need these commands, beginning within the host system’s command line:

Connect to the container's bash (find out the name with 'docker ps')

Courtesy of Ralf Ueberfuhr

docker exec -it --user=root <name-of-kafka-ui-container> /bin/sh

Now, we are connected to the container's bash.

Let's install 'socat'

apk add socat

Use socat to create the port forwarding

socat tcp-listen:9092,fork tcp:host.docker.internal:9092

This will lead to a running process that will not be killed as long as the container is running

image-20240708-171752.png

 

kafka_producer app cheat sheet

Commands

# -t <topic name> # -k <key id> # -v <payload data as a string> # -c <behave as a consumer>, -c -p will cause -c to assume the role # -p <behave as a producer>, -p -c will cause -p to assume the role # -d <number> delay sending or forwarding a message for <number seconds> # -r <number> send or forward the same message <number> times # -f <forward topic> forward messages from <watched topic> specied with -t <watched topic> to <forward topic>, if <forward topic> does not exist, it will be created # Examples java -jar .\target\kafka_demo.jar -p -t celestial-stream -k test1 -v "some message" java -jar .\target\kafka_demo.jar -p -t celestial-stream -k test1 -v "some message" -d 10 java -jar .\target\kafka_demo.jar -p -t celestial-stream -k test1 -v "some message" -r 3 java -jar .\target\kafka_demo.jar -c -t celestial-stream

Related content