Unable to access kafka brokers externally after using LoadBalancer

[ishantanu]

Hi,

I tried exposing the individual Kafka brokers with NodePort and using LoadBalancer as well which created load balancers for each broker. But I am still not able to access them from outside.

Here's what I am seeing in logs:

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:547)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at kafka.network.Processor.poll(SocketServer.scala:575)
	at kafka.network.Processor.run(SocketServer.scala:492)
	at java.lang.Thread.run(Thread.java:748)
[2018-10-04 20:43:35,184] WARN [SocketServer brokerId=0] Unexpected error from /172.20.50.156; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:547)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
	at kafka.network.Processor.poll(SocketServer.scala:575)
	at kafka.network.Processor.run(SocketServer.scala:492)
	at java.lang.Thread.run(Thread.java:748)

Also, is there any other way to expose individual Kafka brokers externally without a need to create a load balancer for each of the brokers?

[solsson]

I've never seen these errors. There are other ways, to be found in various issues. For example #187 comes to mind.

[ishantanu]

@solsson I tried changing InternalIP to ExternalIP but it's not working. I just want to be able to connect my services which are outside of k8s cluster to Kafka which is running on the k8s cluster.

[solsson]

This is a complicated topic and success factors depend a lot on your particular cluster setup. You'll have to dive into the reported errors.

[ishantanu]

I am not sure why I am getting an internalIP value in pod description after changing InternalIP to ExternalIP. Can you point to some issue with this?

[solsson]

There's no magic here. The current init.sh script in the applied config map takes effect whenever you restart a kafka pod.

[ishantanu]

@solsson NVM, got the external access working with NodePorts and modifying security groups. Thanks for the help, closing the issue.