Troubleshooting Kafka DNS Resolution from a Local .NET Application

This blog explains a DNS resolution error that can occur when testing an event-driven .NET application locally with Kafka running in a kind cluster. You’ll learn what causes the issue and how to apply a practical workaround to get your local development environment working quickly.

Pre-requisites:

• A Kafka cluster running on a local Kind cluster.

• Strimzi configured to manage the Kafka cluster.

• A .NET API capable of creating or publishing to Kafka.

• kubectl configured to access your Kind cluster.

• PowerShell running on Windows.

• The BootstrapServers value is intentionally set to, localhost:9092 in appsettings.json

1. First, run the following command to port-forward the Kafka bootstrap service to your local machine. The port forwarding makes Kafka's bootstrap service available to the Windows host at, localhost:9092

kubectl -n kafka port-forward svc/<<kafka-bootstrap-service-name>> 9092:9092

Keep this terminal running.

2. Open another terminal and start the API from the workspace root using below commands.

For example, if the application is located in OrderService folder, do as below

    cd OrderService

    dotnet run

3. In my sample application, the API listens on port 5114, so I use the same port in the scripts throughout this blog.

Open PowerShell and call the API to create an order.

Invoke-RestMethod `
  -Method Post `
  -Uri "http://localhost:5114/orders" `
  -ContentType "application/json" `
  -Body '{"orderId":"ord-1001","product":"Keyboard","quantity":2,"price":49.99}'

The expectation is that the API creates the order successfully and publishes the corresponding Kafka message. Instead, you may see an error similar to below.

Failed to resolve 'phkafkacluster-phkafkacluster-nodepool-0.phkafkacluster-kafka-brokers.kafka.svc:9092':
No such host is known.

Why does this happen?

The .NET API is running directly on the Windows host, while Kafka is running inside the Kind Kubernetes cluster.

The .NET Kafka client initially connects to the Kafka bootstrap service through the port-forward:

    localhost:9092

However, after connecting to the bootstrap service, the Kafka client requests metadata about the Kafka cluster.

Kafka then returns the addresses of the brokers.

Because the Kafka cluster is configured with an internal Kubernetes listener, the broker address returned by Kafka may look like this:

    phkafkacluster-phkafkacluster-nodepool-0.phkafkacluster-kafka-brokers.kafka.svc:9092

This hostname is intended to be resolved from within the Kubernetes network.

Since the .NET application is running directly on Windows and not inside Kubernetes, Windows cannot resolve this Kubernetes DNS hostname.

As a result, the Kafka client fails with the DNS resolution error.

Immediate Workaround

As a quick workaround for local development, we can add the broker hostname to the Windows hosts file and map it to 127.0.0.1. This allows the Windows Kafka client to resolve the Kubernetes broker hostname locally

* Make sure broker port-forward is still running. (ex: kubectl -n kafka port-forward svc/phkafkacluster-kafka-bootstrap 9092:9092)

* Open PowerShell as Administrator and run below commands to add host mapping on Windows

Add the broker hostname:

Add-Content -Path C:/Windows/System32/drivers/etc/hosts `
  -Value "127.0.0.1 <Kafka broker's Kubernetes DNS hostname>"

ex: Add-Content -Path C:/Windows/System32/drivers/etc/hosts `
  -Value "127.0.0.1 phkafkacluster-phkafkacluster-nodepool-0.phkafkacluster-kafka-brokers.kafka.svc"

* Run ipconfig /flushdns

* Start the application again and call api as explain in step2 and 3.


* Under Kaffka, you would be able to see the new topic created successfully.