Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kcat with SSL fails #418

Open
britoaldan opened this issue Feb 14, 2023 · 1 comment
Open

Kcat with SSL fails #418

britoaldan opened this issue Feb 14, 2023 · 1 comment

Comments

@britoaldan
Copy link

we have a kafka cluster with latest images, enabled with tls(ssl).
kafka-conosle producer and consumer works fine.
whereas the kcat utility fails to get the metadata information

kafkacat -b xx.xx.xx.xx:9093 -X security.protocol=SSL -X ssl.ca.location=ca.pem -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure

Note: client auth is diabled and ssl.endpoint.algorithm is none
version of kcat 1.5.0/1.7.0
openssl version: 1.1.1-f
librdkafa version: 1.8.2
confluent kafka version: 7.2.0

Error at kcat :
kafkacat -b xx.xx.xx.xx:9093 -X security.protocol=SSL -X ssl.ca.location=ca.pem -L % ERROR: Failed to acquire metadata: Local: Broker transport failure

Error trace at kafka broker
`{"type":"log", "host":"test-kafka-0.default", "level":"INFO", "systemid":"kafka-98aefcdc873b4bbe80ca61a6728eb4ac", "system":"kafka", "time":"2023-01-13T04:08:07.397", "timezone":"UTC", "log":{"message":"data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-1 - org.apache.kafka.common.network.Selector - [SocketServer listenerType=ZK_BROKER, nodeId=1001] Failed authentication with /172.17.0.23 (channelId=172.17.0.21:9092-172.17.0.23:45010-2) (SSL handshake failed)"}}
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.187 UTC|null:-1|Ignore unknown or unsupported extension (
"session_ticket (35)": {

}
)
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.187 UTC|null:-1|Ignore unknown or unsupported extension (
"encrypt_then_mac (22)": {

}
)
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.188 UTC|null:-1|Ignore unknown or unsupported extension (
"client_certificate_type (21)": {
0000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0090: 00 00 00 ...
}
)
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consuming ClientHello handshake message (
"ClientHello": {
"client version" : "TLSv1.2",
"random" : "1C 5B A3 37 09 DD 1F C4 1D E1 1E DF 5C 33 71 61 E9 0A 23 D6 8C 71 24 23 55 9F D6 B4 41 E6 91 CB",
"session id" : "95 19 AB 65 0A BB 37 A1 21 B4 D7 A7 EB 5F 7F 5C EB 52 38 01 F9 59 E0 61 02 0E 39 AC BA 1A DC A1",
"cipher suites" : "[TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_128_CCM_SHA256(0x1304), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(0xC030), TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA9), TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA8), TLS_ECDHE_ECDSA_WITH_AES_256_CCM(0xC0AD), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(0xC02F), TLS_ECDHE_ECDSA_WITH_AES_128_CCM(0xC0AC), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256(0xC023), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(0xC027), TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA(0xC00A), TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA(0xC014), TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA(0xC009), TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA(0xC013), TLS_RSA_WITH_AES_256_GCM_SHA384(0x009D), TLS_RSA_WITH_AES_256_CCM(0xC09D), TLS_RSA_WITH_AES_128_GCM_SHA256(0x009C), TLS_RSA_WITH_AES_128_CCM(0xC09C), TLS_RSA_WITH_AES_256_CBC_SHA256(0x003D), TLS_RSA_WITH_AES_128_CBC_SHA256(0x003C), TLS_RSA_WITH_AES_256_CBC_SHA(0x0035), TLS_RSA_WITH_AES_128_CBC_SHA(0x002F), TLS_DHE_RSA_WITH_AES_256_GCM_SHA384(0x009F), TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256(0xCCAA), TLS_DHE_RSA_WITH_AES_256_CCM(0xC09F), TLS_DHE_RSA_WITH_AES_128_GCM_SHA256(0x009E), TLS_DHE_RSA_WITH_AES_128_CCM(0xC09E), TLS_DHE_RSA_WITH_AES_256_CBC_SHA256(0x006B), TLS_DHE_RSA_WITH_AES_128_CBC_SHA256(0x0067), TLS_DHE_RSA_WITH_AES_256_CBC_SHA(0x0039), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
"compression methods" : "00",
"extensions" : [
"server_name (0)": {
type=host_name (0), value=test-kafka-0.test-kafka-headless.default.svc.cluster.local
},
"ec_point_formats (11)": {
"formats": [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
},
"supported_groups (10)": {
"versions": [x25519, secp256r1, x448, secp521r1, secp384r1]
},
"session_ticket (35)": {

},
"encrypt_then_mac (22)": {
  
},
"extended_master_secret (23)": {
  <empty>
},
"signature_algorithms (13)": {
  "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, ed25519, ed448, rsa_pss_pss_sha256, rsa_pss_rsae_sha256, rsa_pss_pss_sha384, rsa_pss_rsae_sha384, rsa_pss_pss_sha512, rsa_pss_rsae_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, ecdsa_sha224, rsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1]
},
"supported_versions (43)": {
  "versions": [TLSv1.3, TLSv1.2]
},
"psk_key_exchange_modes (45)": {
  "ke_modes": [psk_dhe_ke]
},
"key_share (51)": {
  "client_shares": [  
    {
      "named group": x25519
      "key_exchange": {
        0000: 24 6E 8A FB D7 CB 18 F0   62 5E 54 42 B9 41 81 81  $n......b^TB.A..
        0010: 49 71 17 82 B9 16 87 68   5C C1 65 DE 2C DC B4 44  Iq.....h\.e.,..D
      }
    },
  ]
},
"client_certificate_type (21)": {
  0000: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0010: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0020: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0030: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0040: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0050: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0060: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0070: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0080: 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00  ................
  0090: 00 00 00                                           ...
}

]
}
)
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: supported_versions
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: psk_key_exchange_modes
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Handling pre_shared_key absence.
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|no server name matchers, ignore server name indication
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: server_name
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: status_request
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: supported_groups
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: ec_point_formats
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: signature_algorithms
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: signature_algorithms_cert
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: status_request_v2
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: extended_master_secret
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: cookie
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unavailable extension: certificate_authorities
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported named group: x25519
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Consumed extension: key_share
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.189 UTC|null:-1|Ignore unsupported extension: renegotiation_info
javax.net.ssl|DEBUG|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|use cipher suite TLS_AES_256_GCM_SHA384
javax.net.ssl|ERROR|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|Fatal (UNEXPECTED_MESSAGE): No common named group (
"throwable" : {
javax.net.ssl.SSLProtocolException: No common named group
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.KeyShareExtension$HRRKeyShareProducer.produce(Unknown Source)
at java.base/sun.security.ssl.SSLExtension.produce(Unknown Source)
at java.base/sun.security.ssl.SSLExtensions.produce(Unknown Source)
at java.base/sun.security.ssl.ServerHello$T13HelloRetryRequestProducer.produce(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.goHelloRetryRequest(Unknown Source)
at java.base/sun.security.ssl.ClientHello$T13ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source)
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:182)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at kafka.network.Processor.poll(SocketServer.scala:1144)
at kafka.network.Processor.run(SocketServer.scala:1047)
at java.base/java.lang.Thread.run(Unknown Source)}

)
javax.net.ssl|WARNING|44|data-plane-kafka-network-thread-1001-ListenerName(SSL)-SSL-2|2023-01-13 04:08:08.190 UTC|null:-1|outbound has closed, ignore outbound application data
`
@britoaldan
Copy link
Author

hi @edenhill
could you please have a look at this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@britoaldan