Multiple Mixcoord components in a single deployment

[nairan-deshaw]

Is it possible to have multiple replicas of a mixcoord in a single Milvus K8s cluster? We're seeing some issues in our mixcoord component and since there is only replica it becomes a single point of failure. When the replicas were increased to 2, the new pod keep crashing with CompareAndSwap error for compare is false for key: querycoord

[yhmo]

Milvus supports Coordinator HA from 2.3.3: https://milvus.io/docs/coordinator_ha.md#Coordinator-HA

Since the coordinators have been merged to mixcoord, we can config the value.yaml like this. And then increase the mixcoord k8s replicas to 2.

mixCoordinator:
  replicas: 2
  activeStandby:
    enabled: true

[nairan-deshaw]

When does the standby kick in? Is it only when k8s reports that the other node is not healthy or is there an internal health check between the two replicas?

[yhmo]

If the active coordinator is down, the standby coord will replace its work after a few seconds.

[nairan-deshaw]

We're facing an issue where the coordinator is timing out on all operations - #36217

The state shows okay, but none of the operations work. We wanted to know if this will be detected by the standby and use for automatic failover?