Deleting non-existent principals leaks provisioning records

[tcsc]

Current behavior:

When an Identity Center principal is marked for deletion, but has already been deleted in AWS, the provisioning system will attempt to delete the principal via SCIM and fail. Due to this failure, the principal's provisioning record remains in the backend with state DELETED.

The provisioning system will continue to attempt to delete this record on every reconciliation pass, but the record itself will never be deleted.

Expected behavior:

Treat deleting a non-existent user or group as a success, and delete the provisioning record.

Bug details:

2024-11-13T15:46:29+11:00 WARN [AWS:IC:PR] Failed provisioning resource plugin_name:aws-identity-center plugin_type:aws-identity-center principal_type:PRINCIPAL_TYPE_ACCESS_LIST principal_id:24d8f4a8-7091-7076-4b20-ea0854129714 error:[
ERROR REPORT:
Original Error: *trace.BadParameterError unexpected status code: 404, detail: GROUP does not exist.
Stack Trace:
	github.com/gravitational/teleport/e/lib/scim/sdk/errors.go:44 github.com/gravitational/teleport/e/lib/scim/sdk.decoreError
	github.com/gravitational/teleport/e/lib/scim/sdk/client.go:472 github.com/gravitational/teleport/e/lib/scim/sdk.(*client).deleteResource
	github.com/gravitational/teleport/e/lib/scim/sdk/client.go:287 github.com/gravitational/teleport/e/lib/scim/sdk.(*client).DeleteGroup
	github.com/gravitational/teleport/e/lib/provisioning/provisioner.go:216 github.com/gravitational/teleport/e/lib/provisioning.(*provisioner).deprovisionPrincipal
	github.com/gravitational/teleport/e/lib/provisioning/provisioner.go:160 github.com/gravitational/teleport/e/lib/provisioning.(*provisioner).Provision
	github.com/gravitational/teleport/e/lib/provisioning/provisioner.go:186 github.com/gravitational/teleport/e/lib/provisioning.(*provisioner).ProvisionAll.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78 golang.org/x/sync/errgroup.(*Group).Go.func1
	runtime/asm_arm64.s:1223 runtime.goexit
User Message: deprovisioning principal
	unexpected status code: 404, detail: GROUP does not exist.] provisioning/provisioner.go:187