Getting Started with Kafka on Kind Cluster

Apache Kafka is one of the most widely used event streaming platforms for building real time data pipelines and streaming applications. Before implementing Kafka in production environments, it is important to understand its core concepts, architecture, and operational model.

A practical way to learn Kafka is by running it locally in a lightweight Kubernetes environment. Kind (Kubernetes in Docker) provides a lightweight Kubernetes environment that can run entirely on a local machine. It is easy to set up and consumes minimal resources.

In this blog, we will be learning how to deploy and run Apache Kafka on a Kind cluster, allowing you to experiment with Kafka concepts and gain hands-on experience in a local development environment.

Prerequisites: Kind installed locally and a single-node Kind cluster.

Let’s install the Strimzi Operator in Kind. Strimzi provides Kubernetes-native automation for Kafka clusters using Kubernetes custom resources. It simplifies deploying, managing, and operating Apache Kafka on Kubernetes.

1. Create a namespace

kubectl create namespace kafka

2. Install the Strimzi operator

kubectl apply -f https://strimzi.io/install/latest?namespace=kafka

3. Run the following command to check the status of the operator

kubectl get pods -n kafka

4. Once the strimzi-cluster-operator is running, deploy a single broker Kafka cluster using the manifest below (singlekafka.yaml).”

Run the following command to apply it.

kubectl apply -f singlekafka.yaml

apiVersion: kafka.strimzi.io/v1
kind: Kafka
metadata:
  name: phkafkacluster
  namespace: kafka
spec:
  kafka:
    version: 4.2.0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
  entityOperator:
    topicOperator: {}
    userOperator: {}

---

apiVersion: kafka.strimzi.io/v1
kind: KafkaNodePool
metadata:
  name: phkafkacluster-nodepool
  namespace: kafka
  labels:
    strimzi.io/cluster: phkafkacluster
spec:
  roles:
    - broker
    - controller
  replicas: 1
  storage:
    type: ephemeral

5. Run the following command to view the status of Kafka

kubectl get pods -n kafka

6. Now we have a Kafka cluster running. Let’s try to access it using another pod called kafka-client

Run following command to create kaffka-client. This command creates a temporary interactive pod in the Kafka namespace using a Kafka client image and opens a Bash shell so we can interact with the Kafka cluster, then deletes the pod when we exit.

kubectl run kafka-client -n kafka -it --rm --image=confluentinc/cp-kafka:7.6.0 -- bash

7. Now we are inside the container. Let’s connect to the Kafka cluster, create a topic, and produce a message.

Create a kaffka topic called test-topic

kafka-topics --bootstrap-server phkafkacluster-kafka-bootstrap:9092 --create --topic test-topic --partitions 1 --replication-factor 1

8. Produce a message

kafka-console-producer --bootstrap-server phkafkacluster-kafka-bootstrap:9092 --topic test-topic

Type the message as shown in the image.

9. Now open a new terminal and create a new kaffka-client01

kubectl run kafka-client01 -n kafka -it --rm --image=confluentinc/cp-kafka:7.6.0 -- bash

10. In the new terminal, run the command below to consume messages.

kafka-console-consumer --bootstrap-server phkafkacluster-kafka-bootstrap:9092 --topic test-topic --from-beginning

Now we know how to run a Kafka cluster in Kind and send and consume messages on it. In the next blog, let’s learn how to access the Kafka dashboard using Kafka UI.