Running Kafka Commands via kubectl exec, work around JMX_PORT conflict

[elm-]

Is anyone executing commands against the cluster from shell scripts? I'm trying to run scripts for topic creation, disk space monitoring, etc. Directly running it from "kubectl exec" fails because of JMX_PORT conflict:

elmar@cpy-salt-master:~$ kubectl --namespace=kafka exec -it kafka-0 -- bin/kafka-topics.sh --zookeeper zookeeper:2181 --list
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 5555; nested exception is: 
	java.net.BindException: Address already in use (Bind failed)
command terminated with exit code 1

The workaround I use is to first bash into the machine and then run the cmds with the JMX_PORT env variable overwritten.

elmar@cpy-salt-master:~$ kubectl --namespace=kafka exec -it kafka-0 bash
root@kafka-0:/opt/kafka# JMX_PORT=4321 bin/kafka-topics.sh --zookeeper zookeeper:2181 --list
[...]
__consumer_offsets
[...]

Unfortunately kubectl exec does not support providing environment variables.

Any ideas how to make this work without first running bash?

[solsson]

I found that inconvenient too at first, after #96 was merged. However, I guess it's bad practice to start extra processes in these containers, in particular if you have memory limits or you monitor the resource footprint of kafka pods.

The shell scripts that come with Kafka can run remotely. The kafka image has likely been pulled already to your nodes, så the following should be rather quick:

kubectl run temp-kafka --image solsson/kafka --rm -ti --command -- bash
# in the attached container
./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --list

If your task needs more preparation I suggest an actual manifest. The example Jobs in #95 is one example.

[elm-]

Thanks, but your suggestion would also require to actually run a shell script after first running bash. So not easy for automation tasks. But you gave me the idea to just run a new instance for the cmds. It's a docker container, so no big overhead, just run instead of bash the actual command. E.g.

kubectl run cmd-kafka --image solsson/kafka --rm -ti --command -- ./bin/kafka-topics.sh --zookeeper zookeeper.kafka:2181 --list

Will test this later.

[elm-]

One change, to have it execute in the namespace, but otherwise this works:

kubectl --namespace=kafka-prod run cmd-kafka --image solsson/kafka --rm -ti --command -- ./bin/kafka-topics.sh --zookeeper zookeeper:2181 --list```