[Bug]: The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.

[iceran]

Bug Description

when i update my strimzi version from 0.20.1 to 0.23.0, pod strimzi-operator cannot run, error log as follow

Steps to reproduce

kl logs -f strimzi-cluster-operator-cd9f59cc4-mqjgb,see the error log as follow

Expected behavior

No response

Strimzi version

0.23.0

Kubernetes version

1.24

Installation method

operator

Infrastructure

aws eks

Configuration files and logs

2024-06-19 10:16:51 INFO Main:122 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2024-06-19 10:16:51 INFO ClusterOperator:81 - Creating ClusterOperator for namespace kafka-qa
2024-06-19 10:16:51 INFO ClusterOperator:98 - Starting ClusterOperator for namespace kafka-qa
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for Kafka operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaMirrorMaker operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaConnect operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaBridge operator
2024-06-19 10:16:52 INFO ClusterOperator:112 - Opened watch for KafkaMirrorMaker2 operator
2024-06-19 10:16:52 WARN WatcherWebSocketListener:96 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-19 10:16:52 WARN WatcherWebSocketListener:96 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-19 10:16:52 ERROR Main:157 - Cluster Operator verticle in namespace kafka-qa failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:98) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Suppressed: java.lang.Throwable: waiting here
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:150) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.WebSocketClientRunner.waitUntilReady(WebSocketClientRunner.java:50) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.waitUntilReady(AbstractWatchManager.java:164) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:809) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:782) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:86) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watchInNamespace(AbstractWatchableResourceOperator.java:44) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:51) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.strimzi.operator.cluster.operator.assembly.AbstractConnectOperator.lambda$createConnectorWatch$2(AbstractConnectOperator.java:211) ~[io.strimzi.cluster-operator-0.23.0.jar:0.23.0]
	at io.strimzi.operator.common.Util.lambda$async$0(Util.java:65) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:179) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:177) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]

Additional context

2024-06-19 10:16:51 INFO Main:122 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2024-06-19 10:16:51 INFO ClusterOperator:81 - Creating ClusterOperator for namespace kafka-qa
2024-06-19 10:16:51 INFO ClusterOperator:98 - Starting ClusterOperator for namespace kafka-qa
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for Kafka operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaMirrorMaker operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaConnect operator
2024-06-19 10:16:51 INFO ClusterOperator:112 - Opened watch for KafkaBridge operator
2024-06-19 10:16:52 INFO ClusterOperator:112 - Opened watch for KafkaMirrorMaker2 operator
2024-06-19 10:16:52 WARN WatcherWebSocketListener:96 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-19 10:16:52 WARN WatcherWebSocketListener:96 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
2024-06-19 10:16:52 ERROR Main:157 - Cluster Operator verticle in namespace kafka-qa failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:98) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Suppressed: java.lang.Throwable: waiting here
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:150) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.WebSocketClientRunner.waitUntilReady(WebSocketClientRunner.java:50) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.waitUntilReady(AbstractWatchManager.java:164) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:809) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:782) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:86) ~[io.fabric8.kubernetes-client-5.3.0.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watchInNamespace(AbstractWatchableResourceOperator.java:44) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:51) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.strimzi.operator.cluster.operator.assembly.AbstractConnectOperator.lambda$createConnectorWatch$2(AbstractConnectOperator.java:211) ~[io.strimzi.cluster-operator-0.23.0.jar:0.23.0]
	at io.strimzi.operator.common.Util.lambda$async$0(Util.java:65) ~[io.strimzi.operator-common-0.23.0.jar:0.23.0]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:179) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:177) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.0.3.jar:4.0.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]

[scholzj]

For a start, please make sure to properly format the logs. Otherwise they are unreadable. You should probably also fix the title to contain something what makes some sense. If you upgraded to Strimzi 0.23, you should make sure that you:

• Did the API conversion according to the docs
• Installed the right versions of the CRDs

Apart from that, this is sooooo old version, that nobody will really remember any details. So not sure how much help you will be able to get apart from these generic pointers.

[iceran]

step1 upgrade kafka,from 2.5.0 to 2.6.0
step2 upgrade k8s version, from 1.21 to 1.24
step3 upgrade strimzi 0.20.1 to strimzi 0.23.0
i have convert crd file and delete the old version , then apply the new version as follow

[iceran]

~/Downloads/api-conversion-0.23.0/bin   master  kl get crd|grep kafka
kafkabridges.kafka.strimzi.io 2024-06-19T08:16:24Z
kafkaconnectors.kafka.strimzi.io 2024-06-19T08:16:50Z
kafkaconnects.kafka.strimzi.io 2024-06-19T08:11:41Z
kafkaconnects2is.kafka.strimzi.io 2024-06-19T10:44:43Z
kafkamirrormaker2s.kafka.strimzi.io 2024-06-19T08:17:27Z
kafkamirrormakers.kafka.strimzi.io 2024-06-19T08:15:48Z
kafkarebalances.kafka.strimzi.io 2024-06-19T08:17:42Z
kafkas.kafka.strimzi.io 2024-06-19T08:09:36Z
kafkatopics.kafka.strimzi.io 2024-06-19T08:13:00Z
kafkausers.kafka.strimzi.io 2024-06-19T08:13:46Z

[iceran]

apiversion is wrong, modify then its ok