Can 2 clusters be deployed in the same namespace

[ThePoipoi]

Hi. Another question for you today, on the possibility to deploy 2 Kafka (Strimzi) clusters on the same namespace.
I've tried, and it seems to be fine, but we always have some errors logged on the entity-operator (topic-operator side).

We always have this periodic error (only on the second cluster entity-operator):

2021-12-09 11:21:05,30301 ERROR [vert.x-eventloop-thread-1] TopicOperator:1490 - Reconciliation #445(periodic -__strimzi-topic-operator-kstreams-topic-store-changelog) KafkaTopic(kafka-a/strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b): Error reconciling KafkaTopic null
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.233.0.1/apis/kafka.strimzi.io/v1beta2/namespaces/kafka-a/kafkatopics. Message: kafkatopics.kafka.strimzi.io "strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=kafka.strimzi.io, kind=kafkatopics, name=strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=kafkatopics.kafka.strimzi.io "strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b" already exists, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={}).

Despite both cluster has created the named topic:

It's like they can't share the same default topics. Do we have something to specify to avoid this ?

[scholzj]

When you run multiple topic and user operators with different Kafka clusters in the same namespace, it will have limitations to what topics and users can exist. Because Kubernetes can have only one KafkaTopic or KafkaUser resource with a given name. So in general, you should run it in different namespaces (or disable the entity operator for one cluster).

[ThePoipoi]

Ha ok, that's why.
So, can you tell me how to disable it ?
And, for my own information, what's the purpose of this entity-operator ? Because everytime I start a cluster, the entity-operator need "a lot" of resources (mem/cpu), but after it's ready, it's always using the same amount of resource, like if it's doing nothing.

[scholzj]

You can disable it in the Kafka CR => in the section .spec.entityOperator where you can just remove the things you want to disable.

[ThePoipoi]

Exact, I've just tried it by my-self and should have avoided this question...
And I've also found an answer for the operator purpose :)

Thanks a lot Jakub !!!

[ThePoipoi]

Hi Jakub, it's me again.

I've just found this part of your documentation: https://strimzi.io/docs/operators/latest/using.html#overview-concepts-topic-operator-str

So, the ClusterOperator is only here to manage K8S resources (KafkaTopic and KafkaUser).
But is it really useful if we don't use those objects ?
I mean, on my project, if we want to create a topic we will directly address Kafka and create it.
If we want to look at consumer, we directly use the kafka-consumer-groups.sh scrip via an exec on Kafka pods.
And so on.

So what will be the problem (if any ?) if we run multiple Kafka clusters on the same namespace ?
For example, I'm thinking of default topics like __consumer_offsets, __strimzi-topic-operator-kstreams-topic-store-changelog and __strimzi_store_topic, are will be created (on Kafka side) even if we disable the TopicOperator ?

Thanks.

[scholzj]

Just to correct you -> the Cluster Operator managed the Kafka clusters (and Connect, Mirror Maker etc.).

The Entity Operator which consists of the User and Topic operators managed the topics and users. Its advantage is that it allows you manage users and topics in a Kube-native declarative way (e.g. using YAML and kubectl). But if you do not want to use that and want to manage your users / topics directly in Kafka, they are optional and you can just disable them.

With regards to the particular topics, __consumer_offsets is created by Kafka itself. The other two topics are internal topics of the Topic Operator. So if you do not use Topic Operator you can just delete them.

[ThePoipoi]

Perfect answer. Thanks a lot ;)

[Ruby0510]

Hi @scholzj!

My case is also that deployed more than one kafka cluster in one namespace.

As you suggested "Because Kubernetes can have only one KafkaTopic or KafkaUser resource with a given name. So in general, you should run it in different namespaces (or disable the entity operator for one cluster)."

So I only enabled entity operator for one kafka cluster and disabled it for others in the namespace.

Then I created one KafkaUser resource which label to the kafka cluster which enabled entity operator.
Here comes the question how to make the given user associated with all the kafka clusters in the namespace? It seems that one kafkauser has be and only can be labeled with one kafka cluster.

With this case, I want to use the same username for the kafka clusters in the namespace. No matter one same user or multiple users with same name.

Could you help point me a feasible solution? Thanks a lot!

BR
Ruby

[scholzj]

You cannot do that. One user can always belong only to one cluster. Also, keep in mind that with the entity operator enabled only in one cluster, only that cluster has managed users and topics by the operator. In the other clusters, you have to do it through the KAfka Admin API directly in Kafka.

[Ruby0510]

Thanks for quick reply!
However, I just found another statement in the document: https://strimzi.io/docs/operators/0.41.0/deploying#deploying-the-user-operator-using-the-cluster-operator-str
"When using multiple User Operators with a single namespace, a user with a given username can exist in more than one Kafka cluster."

But to be honest, I am confused. Isn't the line saying that it's possible to enable entity(user) operator for more than one kafka clusters in one namespace? And then one given user could exist in the multiple kafka clusters?

[scholzj]

I'm not sure what that is trying to say (cc @PaulRMellor). You can of course have a user with the same name in every cluster in the namespace. But only in one of them it will be managed by the KafkaUser resource. The rest will need to be created by you using Kafka Admin API.

But in short, the recommendation is not have multiple Kafka clusters in the same namespace. That helps you to avoid all these issues.

[Ruby0510]

@scholzj Yes, I understand the recommendation is about best practice. However we have limitation. If there is no good way to get help with KafkaUser under my case, I have to go to kafka API. Thanks for your kind help.

Meanwhile, I am still curious about the description in document. Looking forward to your answer @PaulRMellor :)

[scholzj]

Meanwhile, I am still curious about the description in document. Looking forward to your answer @PaulRMellor :)

One way or another, it is as I told you. So this is more about to clarify the docs and fix them if needed.