Exposing Kafka from kind to a Local Windows Application

In our previous blog post, we discussed a workaround for the DNS error that occurs when an application running on a local Windows machine connects to Kafka. In this post, we will discuss the proper solution.

Our Kafka cluster runs inside Docker through a local kind Kubernetes cluster rather than directly on the host machine. Kafka advertises broker addresses using Kubernetes DNS names, which can be resolved from inside the cluster but not by an application running on Windows. To make Kafka reachable from the host, we must expose the required ports from the kind node container to the host machine.

Pre-requisites: 

• A Kafka cluster running on a local Kind cluster.

• Strimzi configured to manage the Kafka cluster.

• A .NET API capable of creating or publishing to Kafka.

• kubectl configured to access your Kind cluster.

• PowerShell running on Windows.

This happens because Kubernetes NodePort is reachable inside the kind node, but not automatically from Windows or macOS. Therefore, a Kubernetes Service may expose NodePorts such as 32092 and 32100, while the host machine cannot connect to those ports directly.

The solution is to configure extraPortMappings when creating the kind cluster. For accessing the Kafka running in a kind cluster from .NET application running on Windows machine, we can use the kind cluster configuration as below.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 32092
        hostPort: 32092
        listenAddress: "127.0.0.1"
        protocol: TCP
      - containerPort: 32100
        hostPort: 32100
        listenAddress: "127.0.0.1"
        protocol: TCP

extraPortMappings tells kind to expose additional ports from the kind node container to the host machine.

containerPort is the port on the kind node container that should be exposed.

hostPort is the corresponding port on the host machine.

Kafka uses one port for the initial bootstrap connection and may use different ports for the broker addresses returned in the metadata. Therefore, Kafka requires multiple port mappings in this sample project.

ex: Under the Listeners section of the Kafka manifest, we have the following port configurations for the bootstrap server and brokers.

    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: external
        port: 9094
        type: nodeport
        tls: false  
        configuration:
          bootstrap:
            nodePort: 32092
          brokers:
            - broker: 0
              nodePort: 32100
              advertisedHost: localhost
              advertisedPort: 32100

Once we have created the kind cluster with the extraPortMappings and Kafka listener configuration, we can test the connection from the host machine using the commands below. Run these commands in separate PowerShell windows.

First, run the following command to port-forward the Kafka bootstrap service:

kubectl port-forward -n kafka service/<kubernetes service for kaffka bootstrap> 32092:9094 --address 127.0.0.1

ex: kubectl port-forward -n kafka service/phkafkacluster-kafka-external-bootstrap 32092:9094 --address 127.0.0.1

In another PowerShell window, run the following command to port-forward the Kafka broker.

kubectl port-forward -n kafka service/<Kubernetes service for kafka broker> 32100:9094 --address 127.0.0.1

ex: kubectl port-forward -n kafka service/phkafkacluster-phkafkacluster-nodepool-0 32100:9094 --address 127.0.0.1

Then, open another PowerShell window and run the command below to test the connection.

Test-NetConnection localhost -Port 32092

Test-NetConnection localhost -Port 32100

If all the configurations are correct, we should see a result similar to the following, with TcpTestSucceeded : True.

Now we can run the .NET application and test the connection. The application should be able to connect to Kafka successfully without encountering any DNS errors