Error connecting to jmx service

[cassanellicarlo]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running master-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Hello,
I was trying to enable metrics for Kafka and view them via kafka-ui. I configured JMX on the kafka broker and on the kafka-ui, but i'm getting an error when i start kafka-ui.

2023-06-13 13:36:24,294 DEBUG [boundedElastic-1] c.p.k.u.s.m.JmxMetricsRetriever: 0 metrics collected for service:jmx:rmi:///jndi/rmi://<hostname>:9999/jmxrmi 2023-06-13 13:36:24,294 DEBUG [boundedElastic-2] c.p.k.u.s.m.JmxMetricsRetriever: 0 metrics collected for service:jmx:rmi:///jndi/rmi://<hostname>:9999/jmxrmi 2023-06-13 13:36:24,293 ERROR [boundedElastic-6] c.p.k.u.s.m.JmxMetricsRetriever: Error connecting to service:jmx:rmi:///jndi/rmi://<hostname>:9999/jmxrmi java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset] at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370) at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.withJmxConnector(JmxMetricsRetriever.java:79) at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.retrieveSync(JmxMetricsRetriever.java:66) at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.lambda$retrieve$0(JmxMetricsRetriever.java:52) at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67) at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: javax.naming.CommunicationException: null at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:138) at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220) at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409) at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839) at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813) at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302) ... 12 common frames omitted Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:308) at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204) at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:344) at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116) at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:134) ... 17 common frames omitted Caused by: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:976) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263) at java.base/java.io.DataInputStream.readByte(DataInputStream.java:271) at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:247) ... 21 common frames omitted 2023-06-13 13:36:24,294 DEBUG [boundedElastic-6] c.p.k.u.s.m.JmxMetricsRetriever: 0 metrics collected for service:jmx:rmi:///jndi/rmi://<hostname>:9999/jmxrmi 2023-06-13 13:36:24,294 DEBUG [boundedElastic-6] c.p.k.u.s.ClustersStatisticsScheduler: Metrics updated for cluster: DC1

Expected behavior

I can start kafka ui and see the metrics in the metrics tab.

Your installation details

kafka-ui:v0.7.0

run kafka -ui via podman, with those additional two environmental variables

--env KAFKA_CLUSTER_0_METRICS_PORT=9999
--env KAFKA_CLUSTER_1_METRICS_PORT=9999

run kafka broker via podman, server.properties file contains the following

# Enable JMX
com.sun.management.jmxremote=true
# Set JMX port
com.sun.management.jmxremote.port=9999
com.sun.management.jmxremote.rmi.port=9999
java.rmi.server.hostname={{ansible_hostname}}

Steps to reproduce

• Configure JMX in kafka broker
• Start kafka-ui with metrics

Screenshots

No response

Logs

No response

Additional context

No response

[Haarolean]

Please provide your full config with the actual hostnames

[cassanellicarlo]

Unfortunately i can't copy the full hostnames.

[cassanellicarlo]

@Haarolean seems the client is trying to connect to localhost instead of the remote jmx server.

[cassanellicarlo]

Problem solved, i had to add the following configuration for kafka container

        - "JMX_PORT=9999"
        - "KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname={{ansible_hostname}} -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=9999"

I was putting them in the server.properties file for kafka but apparently they are not loaded.