Not possible to manage KMS key and Event Streams instance using it in same terraform state

[ocofaigh]

Due to the way the Event Streams service handles de-registration of an instance from a KMS root key, it make it impossible to manage the KMS key and the Event Streams instance in the same terraform state.

Here is how the Event Streams service handles de-registration:

• When delete of instance occurs, it goes into the reclamation list
• Only when that reclamation expires (or if someone forcefully deletes the reclamation) does the re-registration of the instance from the key occur. NOTE: The auth policy must be still in place at this point for de-registration to occur.

Due to this it makes it impossible to manage the key, auth policy and the instance in the same terraform.

In the Terraform Configuration Files section below, I have included a code snippet that can be used to verify this. If you do terraform apply and then terraform destroy on it, the destroy failed with this:

│ Error: [ERROR] Error while deleting: kp.Error: correlation_id='f57445b8-8886-429c-9f44-ba130043b868', msg='Conflict: Key could not be deleted: Please see `reasons` for more details (PREV_KEY_DEL_ERR)', reasons='[PREV_KEY_DEL_ERR: The key cannot be deleted because it's protecting a cloud resource that has a retention policy: Before you delete this key, contact an account owner to remove the retention policy on each resource that is associated with the key - FOR_MORE_INFO_REFER: https://cloud.ibm.com/docs/key-protect?topic=key-protect-delete-keys#delete-key-force]'. The key has the following active registrations which may interfere with deletion: [crn:v1:bluemix:public:messagehub:us-south:a/abac0df06b644a9cabc6e44f55b3880e:e62e3d5d-93e1-4a8a-acc7-a5113d093adc::]
│ 
│ ---
│ id: terraform-a05cda1f
│ summary: '[ERROR] Error while deleting: kp.Error: correlation_id=''f57445b8-8886-429c-9f44-ba130043b868'',
│   msg=''Conflict: Key could not be deleted: Please see `reasons` for more details
│   (PREV_KEY_DEL_ERR)'', reasons=''[PREV_KEY_DEL_ERR: The key cannot be deleted because
│   it''s protecting a cloud resource that has a retention policy: Before you delete
│   this key, contact an account owner to remove the retention policy on each resource
│   that is associated with the key - FOR_MORE_INFO_REFER: https://cloud.ibm.com/docs/key-protect?topic=key-protect-delete-keys#delete-key-force]''.
│   The key has the following active registrations which may interfere with deletion:
│   [crn:v1:bluemix:public:messagehub:us-south:a/abac0df06b644a9cabc6e44f55b3880e:e62e3d5d-93e1-4a8a-acc7-a5113d093adc::]'
│ severity: error
│ resource: ibm_kms_key
│ operation: delete
│ component:
│   name: github.com/IBM-Cloud/terraform-provider-ibm
│   version: 1.66.0
│ ---

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

tf 1.6.6
ibm provider 1.66.0

Affected Resource(s)

• ibm_resource_instance

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "ibm" {
  ibmcloud_api_key = var.ibmcloud_api_key
  region           = "us-south"
}

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    ibm = {
      source  = "IBM-Cloud/ibm"
      version = "1.66.0"
    }
  }
}

# Lookup Default resource group ID
data "ibm_resource_group" "resource_group" {
  name  = "Default"
}

# Create Key Protect instance
resource "ibm_resource_instance" "key_protect_instance" {
  name              = "my-key-protect"
  resource_group_id = data.ibm_resource_group.resource_group.id
  service           = "kms"
  plan              = "tiered-pricing"
  location          = "us-south"
  tags              = []
  parameters = {
    allowed_network : "public-and-private"
  }
}

# create key ring
resource "ibm_kms_key_rings" "key_ring" {
  endpoint_type = "public"
  instance_id   = ibm_resource_instance.key_protect_instance.id
  key_ring_id   = "my-key-ring"
  force_delete  = true
}

# create key
resource "ibm_kms_key" "key" {
  depends_on    = [ibm_iam_authorization_policy.kms_policy]
  instance_id   = ibm_resource_instance.key_protect_instance.id
  key_name      = "my-key"
  key_ring_id   = ibm_kms_key_rings.key_ring.key_ring_id
  standard_key  = false
  endpoint_type = "public"
  force_delete  = true
}

# Create s2s IAM authorization policy to allow messagehub to access KMS for the encryption key
resource "ibm_iam_authorization_policy" "kms_policy" {
  source_service_name         = "messagehub"
  source_resource_group_id    = data.ibm_resource_group.resource_group.id
  target_service_name         = "kms"
  target_resource_instance_id = ibm_resource_instance.key_protect_instance.id
  roles                       = ["Reader"]
}

# create Event Streams instance and encrypt it with Key Protect key
resource "ibm_resource_instance" "es_instance" {
  depends_on        = [ibm_iam_authorization_policy.kms_policy]
  name              = "my-event-streams"
  service           = "messagehub"
  plan              = "enterprise-3nodes-2tb"
  location          = "us-south"
  resource_group_id = data.ibm_resource_group.resource_group.id
  tags              = []
  timeouts {
    create = "3h"
    update = "1h"
    delete = "15m"
  }

  parameters = {
    service-endpoints = "public-and-private"
    throughput        = "150"
    storage_size      = "2048"
    kms_key_crn       = ibm_kms_key.key.crn
  }
}

Debug Output

Panic Output

Expected Behavior

The use case is a standard pattern used by many other services - so I expect it to also work for Event Streams

Actual Behavior

Cannot manage KMS key and Event Streams in same terraform

Steps to Reproduce

1. terraform apply
2. terraform destroy

Important Factoids

References

• #0000

[stephaniegalang]

Hi there!

This error is returned when a service registers an association with a given Key Protect key AND requests that deletion (even via force) be disallowed. This feature is intended to prevent users from violating government and/or company mandated data retention guarantees as ensured by COS bucket retention policies. In a scenario where data retention is mandated, this error is desirable and should not be subverted.

Unless Event Streams has any such data guarantee, they should not be requesting this behavior even if this means allowing users to accidentally render data within an Event Stream instance unusable or un-reclaimable.

One solution for this particular ticket would be for the Event Streams service to discontinue enablement of "preventKeyDeletion" during registration against a Key Protect key. If this is not acceptable, the Event Streams TF destroy behavior can be updated to delete the reclamation resulting from the instance deprovision.