Deployments getting randomly deleted without any changes in the GIT REPO

[apenmetsa-conga]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We are experiencing random deletion of deployments from the Kubernetes cluster without any changes being made to the GIT Repo. This behaviour has been noticed multiple times in the last couple of days.
The deployments are again restored after re-importing the cluster multiple times to Fleet.
This has already caused significant outage to our Production environments.

Expected Behavior

Ideally, if there is no change in the GIT Repos, the deployments should not be impacted. The deployment should also not be impacted even when we re-import the cluster or do a force update multiple times.
However, in the current scenario, the deployments are being deleted randomly.

Steps To Reproduce

This is a Random behaviour observed. So, there are no definite steps to reproduce this.

Environment

- Architecture: arm64
- Fleet Version: 0.9.6
- Cluster:
  - Provider: AWS EKS Cluster
  - Options: We have observed this behaviour on multiple clusters. Some of the clusters had 8 nodes. 
  - Kubernetes Version: We have observed this behaviour on multiple clusters. Some of them were on 1.29 and some of them were on 1.27

Logs

No response

Anything else?

No response

[kkaempf]

Please provide logs or instructions how to replicate your issue.

[apenmetsa-conga]

Logs

I have updated the logs. Can you please check

`

Click to expand

I0719 14:45:11.961742 1 leaderelection.go:250] attempting to acquire leader lease cattle-fleet-system/fleet-agent-lock...
I0719 14:45:57.524040 1 leaderelection.go:260] successfully acquired lease cattle-fleet-system/fleet-agent-lock
time="2024-07-19T14:45:57Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2024-07-19T14:45:57Z" level=info msg="Creating clusterregistration with id '88jssscnzt6dcpg7l967zrdxp4n4fv7fpr7k84k6r5ncbc6qjrbgvn' for new token"
time="2024-07-19T14:45:59Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:01Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:03Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:05Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:07Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:09Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:11Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:13Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2' on management cluster for request 'fleet-default/request-xlkdx': secrets "c-de90535e0ef4e2322ca1ac3c7d5c22d8f89630298108e2ffa61c1662135e2" not found"
time="2024-07-19T14:46:15Z" level=error msg="Failed to register agent: registration failed: new client config cannot list bundledeployments on management cluster: bundledeployments.fleet.cattle.io is forbidden: User "system:serviceaccount:cluster-fleet-default-c-ncbdj-fc809d6b1ea0:request-xlkdx-7896948d-bd11-4bbd-a02c-45d97533a276" cannot list resource "bundledeployments" in API group "fleet.cattle.io" in the namespace "cluster-fleet-default-c-ncbdj-fc809d6b1ea0""
time="2024-07-19T14:47:15Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2024-07-19T14:47:16Z" level=info msg="Creating clusterregistration with id '88jssscnzt6dcpg7l967zrdxp4n4fv7fpr7k84k6r5ncbc6qjrbgvn' for new token"
time="2024-07-19T14:47:18Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:20Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:22Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:24Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:26Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:28Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:30Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:32Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f' on management cluster for request 'fleet-default/request-vxs49': secrets "c-5c2c05c4cefd8fd7e4d5807456b7882e1abcac572408d6e454b8c41972c8f" not found"
time="2024-07-19T14:47:34Z" level=error msg="Failed to register agent: registration failed: new client config cannot list bundledeployments on management cluster: bundledeployments.fleet.cattle.io is forbidden: User "system:serviceaccount:cluster-fleet-default-c-7tbsk-492dbca4cbdc:request-vxs49-ea6b0a2f-a1e6-4ef2-bfb5-64d02a021c61" cannot list resource "bundledeployments" in API group "fleet.cattle.io" in the namespace "cluster-fleet-default-c-7tbsk-492dbca4cbdc""
time="2024-07-19T14:48:34Z" level=warning msg="Cannot find fleet-agent secret, running registration"
time="2024-07-19T14:48:34Z" level=info msg="Creating clusterregistration with id '88jssscnzt6dcpg7l967zrdxp4n4fv7fpr7k84k6r5ncbc6qjrbgvn' for new token"
time="2024-07-19T14:48:36Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1' on management cluster for request 'fleet-default/request-v9f7n': secrets "c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1" not found"
time="2024-07-19T14:48:38Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1' on management cluster for request 'fleet-default/request-v9f7n': secrets "c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1" not found"
time="2024-07-19T14:48:40Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1' on management cluster for request 'fleet-default/request-v9f7n': secrets "c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1" not found"
time="2024-07-19T14:48:42Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1' on management cluster for request 'fleet-default/request-v9f7n': secrets "c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1" not found"
time="2024-07-19T14:48:44Z" level=info msg="Waiting for secret 'cattle-fleet-clusters-system/c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1' on management cluster for request 'fleet-default/request-v9f7n': secrets "c-79f3e431b28eb993e4da257ad56460537048bf2e96441e79cb80f28e95eb1" not found"
time="2024-07-19T14:49:02Z" level=error msg="failed to report cluster node status: Unauthorized"
`

[apenmetsa-conga]

Hello,
Can we get any updates on this.. ?