Multiple JMX ports | three brokers #3890

codebased · 2023-06-01T09:06:32Z

codebased
Jun 1, 2023

Hey Guys,
I have got three brokers in one docker-compose.yml

Each broker is exposing 9997 jmx port as 9997 , 9998, 9999.

broker1:
    image: confluentinc/cp-kafka
    hostname: broker1
    container_name: broker1
    ports:
      - "9092:9092"
      - "9997:9997"
    restart: always
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://broker1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_NODE_ID: 1
      KAFKA_JMX_PORT: 9997
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka0 -Dcom.sun.management.jmxremote.rmi.port=9997
    volumes:
      - ./logs/broker1/data:/var/lib/kafka/data
    depends_on:
      - zoo1

  broker2:
    image: confluentinc/cp-kafka
    hostname: broker2
    container_name: broker2
    ports:
      - "9093:9093"
      - "9998:9997"
    restart: always
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://broker2:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 2
      KAFKA_NODE_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_JMX_PORT: 9997
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka0 -Dcom.sun.management.jmxremote.rmi.port=9997
    volumes:
      - ./logs/broker2/data:/var/lib/kafka/data
    depends_on:
      - zoo1
    
  broker3:
    image: confluentinc/cp-kafka
    container_name: broker3
    hostname: broker3
    ports:
      - "9094:9094"
      - "9999:9997"
    restart: always
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://broker3:19094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 3
      KAFKA_NODE_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_JMX_PORT: 9997
      KAFKA_JMX_HOSTNAME: localhost
      KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kafka0 -Dcom.sun.management.jmxremote.rmi.port=9997
    volumes:
      - ./logs/broker3/data:/var/lib/kafka/data
    depends_on:
      - zoo1

Because my Kafka ui configuration is searching for 9997, the rest two brokers are not reachable for kafka-ui

kafka:
  clusters:
    -
      name: local
      bootstrapServers: broker1:19092
      schemaRegistry: http://schema-registry:8081
      # schemaRegistryAuth:
      #   username: username
      #   password: password
#     schemaNameTemplate: "%s-value"
      metrics:
        port: 9997
        type: JMX

Error:

ollection JMX metrics 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui 2023-06-01 19:05:13 kafka-ui for service:jmx:rmi:///jndi/rmi://broker2:9997/jmxrmi
| 2023-06-01 09:05:13,540 ERROR [boundedElastic-1] c.p.k.u.s.m.JmxMetricsRetriever: Error connecting to service:jmx:rmi:///jndi/rmi://broker1:9997/jmxrmi
| java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
| java.net.ConnectException: Connection refused
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:626)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:217)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
| at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
| at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.withJmxConnector(JmxMetricsRetriever.java:79)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.retrieveSync(JmxMetricsRetriever.java:66)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.lambda$retrieve$0(JmxMetricsRetriever.java:52)
| at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67)
| at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
| at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
| at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
| at java.base/java.lang.Thread.run(Thread.java:833)
| Caused by: java.net.ConnectException: Connection refused
| at java.base/sun.nio.ch.Net.connect0(Native Method)
| at java.base/sun.nio.ch.Net.connect(Net.java:579)
| at java.base/sun.nio.ch.Net.connect(Net.java:568)
| at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
| at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
| at java.base/java.net.Socket.connect(Socket.java:639)
| at java.base/java.net.Socket.connect(Socket.java:588)
| at java.base/java.net.Socket.(Socket.java:512)
| at java.base/java.net.Socket.(Socket.java:292)
| at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
| ... 18 common frames omitted
| 2023-06-01 09:05:13,553 DEBUG [boundedElastic-1] c.p.k.u.s.m.JmxMetricsRetriever: 0 metrics collected for service:jmx:rmi:///jndi/rmi://broker1:9997/jmxrmi
| 2023-06-01 09:05:13,553 ERROR [boundedElastic-3] c.p.k.u.s.m.JmxMetricsRetriever: Error connecting to service:jmx:rmi:///jndi/rmi://broker2:9997/jmxrmi
| java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
| java.net.ConnectException: Connection refused
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:626)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:217)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
| at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
| at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.withJmxConnector(JmxMetricsRetriever.java:79)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.retrieveSync(JmxMetricsRetriever.java:66)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.lambda$retrieve$0(JmxMetricsRetriever.java:52)
| at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67)
| at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
| at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
| at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
| at java.base/java.lang.Thread.run(Thread.java:833)
| Caused by: java.net.ConnectException: Connection refused
| at java.base/sun.nio.ch.Net.connect0(Native Method)
| at java.base/sun.nio.ch.Net.connect(Net.java:579)
| at java.base/sun.nio.ch.Net.connect(Net.java:568)
| at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
| at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
| at java.base/java.net.Socket.connect(Socket.java:639)
| at java.base/java.net.Socket.connect(Socket.java:588)
| at java.base/java.net.Socket.(Socket.java:512)
| at java.base/java.net.Socket.(Socket.java:292)
| at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
| ... 18 common frames omitted
| 2023-06-01 09:05:13,555 DEBUG [boundedElastic-3] c.p.k.u.s.m.JmxMetricsRetriever: 0 metrics collected for service:jmx:rmi:///jndi/rmi://broker2:9997/jmxrmi
| 2023-06-01 09:05:13,563 ERROR [boundedElastic-2] c.p.k.u.s.m.JmxMetricsRetriever: Error connecting to service:jmx:rmi:///jndi/rmi://broker3:9997/jmxrmi
| java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
| java.net.ConnectException: Connection refused
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:626)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:217)
| at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:204)
| at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
| at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
| at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.withJmxConnector(JmxMetricsRetriever.java:79)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.retrieveSync(JmxMetricsRetriever.java:66)
| at com.provectus.kafka.ui.service.metrics.JmxMetricsRetriever.lambda$retrieve$0(JmxMetricsRetriever.java:52)
| at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67)
| at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
| at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
| at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
| at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
| at java.base/java.lang.Thread.run(Thread.java:833)
| Caused by: java.net.ConnectException: Connection refused
| at java.base/sun.nio.ch.Net.connect0(Native Method)
| at java.base/sun.nio.ch.Net.connect(Net.java:579)
| at java.base/sun.nio.ch.Net.connect(Net.java:568)
| at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:588)
| at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
| at java.base/java.net.Socket.connect(Socket.java:639)
| at java.base/java.net.Socket.connect(Socket.java:588)
| at java.base/java.net.Socket.(Socket.java:512)
| at java.base/java.net.Socket.(Socket.java:292)
| at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
| at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:620)
| ... 18 common frames omitted

Answered by Haarolean

Jun 5, 2023

Hi, the solution is simple: don't expose JMX ports for each container, and specify container hostnames in k-ui's bootstrapServers.

View full answer

Haarolean · 2023-06-05T12:11:32Z

Haarolean
Jun 5, 2023

Hi, the solution is simple: don't expose JMX ports for each container, and specify container hostnames in k-ui's bootstrapServers.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple JMX ports | three brokers #3890

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Multiple JMX ports | three brokers #3890

codebased Jun 1, 2023

Replies: 1 comment

Haarolean Jun 5, 2023

codebased
Jun 1, 2023

Haarolean
Jun 5, 2023