Removing otomi from the cluster

[j-zimnowoda]

Description

Removing resources deployed by otomi is not a trivial issue and it turns out that removing helm releases is not always sufficient.

Upon succesfull otomi destroy execution there are many resources that are left over. These are :

• namespaces
• CRDs
• resources created by operators (happens randomly)

Procedure for removing left over resources

To wipe out otomi from the cluster I performed the following steps.

Removing namespaces and resource that are not managed by helm by still created by otomi:

hf -f helmfile.tpl/helmfile-init.yaml template | kubectl delete -f -
kubectl delete -f charts/prometheus-operator/crds

The namespaces stuck in Terminating state due to finalisers that blocks it. I had to remove finalisers to unblock namespace removal.

for ns in $(kubectl get ns --field-selector status.phase=Terminating -o jsonpath='{.items[*].metadata.name}'); do  kubectl get ns $ns -ojson | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f -; done

Removing CRDs:

k delete `k get crd  -o name |grep  istio`
k delete `k get crd -o name |grep  vault` 
k delete `k get crd -o name |grep  knative`
k delete `k get crd -o name |grep  kubeapps` 
k delete `k get crd -o name |grep  gatekeeper` 
k delete `k get crd -o name |grep  'coreos.com'` 
k delete `k get crd -o name |grep  'kubernetes-client.io'` 
k delete `k get crd -o name |grep  'monitoring.coreos.com'` 

Webhooks

Check if there are no Webhooks from istio/gatekeeper etc.

k get mutatingwebhookconfigurations 
k get validatingwebhookconfigurations.admissionregistration.k8s.io    

Zombie resources

If operator artefacts are deleted then operator needs some time to remove all related resources.
Unfortunately, the otomi destroy does not wait, thus some zombie Webhooks or other resource may be left in the cluster.

[Dunky13]

Would the process become easier if all namespaces that are created through otomi are prepended with something like otomi-?
So you can remove anything that has otomi-*?

[0-sv]

I think this suggestion would be a use-case for k8s labels.

[Morriz]

Please understand that ALL of those resources must be removed manually and should NOT be destroyed automatically.

There is NO use case for talking about this, other than one that simply violates k8s best practices.

[Morriz]

Also, if operators don't clean up well after themselves, then please file a bug with that app. What operators don't do that correctly @j-zimnowoda ?

[j-zimnowoda]

It happened to me that istio did not remove mutating Webhooks.
But I am not sure if this is a problem with operator itself or maybe kubernetes that after graceful period evicts operator Pod

[j-zimnowoda]

There is NO use case for talking about this, other than one that simply violates k8s best practices.

The problem is that this practices are not yet established. For example what to do with CRD, it is obviously helm issue.
Anyway I believe we should at least know what is left over. Otherwise we may stumble on very weird issues that are caused by some dangling resources.

[Morriz]

If you want to introduce an extra parameter to destroy that removes ALL resources for dev purposes then go ahead ;)

[j-zimnowoda]

oki, I will create a user story and you will be able to prioritise it. thanks for response