Error Docker Swarm with Debezium Connector

[hieu29791]

Hi, when I set up my Swarm with this stack: Kafka(multi-broker), zookeeper, debezium. Kafka and zookeeper are working, can create topic, consumer and producer, but debezium show error: org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties. I'm not modifying anything, just default config as docker-stack below:

version: '3.6'
services:
   zoo:
      image: wurstmeister/zookeeper
      ports:
         - '2181:2181'
      volumes:
         - zoo-data:/tmp/zookeeper
      deploy:
         replicas: 1
         placement:
            constraints:
               - node.labels.type==zoo
   kafka:
      image: wurstmeister/kafka:latest
      ports:
         - target: 9094
           published: 9094
           protocol: tcp
           mode: host
      environment:
         HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
         KAFKA_ZOOKEEPER_CONNECT: zoo:2181
         KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
         KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
         KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
         KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
         #KAFKA_CREATE_TOPICS: "Topic1:1:2,Topic2:1:1:compact"
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - kafka-data:/tmp/kafka-logs
      deploy:
         mode: global
         placement:
            constraints:
               - node.labels.name==kafka
      depends_on:
         - zoo

   debezium:
      image: debezium/connect:0.8
      hostname: connect
      ports:
         - '8083:8083'
      environment:
         BOOTSTRAP_SERVERS: kafka:9094
         GROUP_ID: 1
         CONFIG_STORAGE_TOPIC: my_connect_configs
         OFFSET_STORAGE_TOPIC: my_connect_offsets
      deploy:
         placement:
            constraints:
               - node.labels.type==dbz
      depends_on:
         - kafka
volumes:
   kafka-data:
   zoo-data:

When I check docker services log debezium, it show error

shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 2018-08-03 04:33:27,034 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:77)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:258)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | 	... 2 more
shippo_kafka_debezium.1.5l1yhz27r6p2@kafka1    | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.

Can anyone show me how to fix this error, I'm new with this stack so, during a few days research, I can't figure out it. Thank you so much !

[sscaling]

I'm not super familiar with swarm, but I don't think you can use global deployment mode + placement constraints (1st google result agrees https://forums.docker.com/t/docker-swarm-constraints-being-ignored/31555/2)

As for connecting, in the bootstrap servers should be the list of unique host:port combinations which is a function of the number of brokers. i.e. if you have 3 kafka brokers kafka-1.internal, kafka-2.internal, i'd expect your bootstrap servers config to be kafka-1.internal:9094,kafka-2.internal:9094

Someone has recently submitted a PR - #377 (not merged) that injects the swarm IP as the OUTSIDE advertised hostname instead of the internal hostname. It might be worth trying that as well.

I know nothing about debezium so can't offer any advice regarding that. However, it's probably worth checking that the standard kafka-console-producer / kafka-console-consumer tools work OK within the cluster before trying to connect with an external component.

[hieu29791]

@sscaling Thanks for your comment, as I said above, my Kafka and Zookeeper working well with this config (mode:global and constraint I will test later, thanks for notice me ^^). I can run kafka-console-producer and kafka-console-consumer with no problem. My problem is when I bring debezium to this stack, debezium not working, kafka and zookeeper working normally. And debezium logs show as above.

[sscaling]

I don't think there's much more help we can provide if the docker image is working correctly.

Try the PR change linked in previous comment (as that will return IP rather than hostname - could be experiencing the same issue that PR is trying to resolve). If that doesn't work try the debezium mailing list / forums / stackoverflow.