Add new modules for required storage/idbroker and credential permissions for AWS

README.md

@@ -10,6 +10,8 @@ This repository contains a number of Terraform modules for creation of the pre-r
 | [terraform-cdp-azure-pre-reqs](modules/terraform-cdp-azure-pre-reqs/README.md) | For all Azure pre-requisite Cloud resources |
 | [terraform-cdp-gcp-pre-reqs](modules/terraform-cdp-gcp-pre-reqs/README.md) | For all GCP pre-requisite Cloud resources |
 | [terraform-cdp-deploy](modules/terraform-cdp-deploy/README.md) | For deployment of CDP on AWS, Azure or GCP. |
+| [terraform-aws-cred-permissions](modules/terraform-aws-cred-permissions/README.md) | Module for creation of the Cross Account Credential pre-requisite on AWS. Note that this module is called from the terraform-cdp-aws-prereqs module. |
+| [terraform-aws-permissions](modules/terraform-aws-permissions/README.md) | Module for creation of the AWS IAM permissions required by the (CDP) Public Cloud environment and datalake deployment. Note that this module is called from the terraform-cdp-aws-prereqs module. |
 | [terraform-aws-vpc](modules/terraform-aws-vpc/README.md) | Module for creation of the VPC networking resources on AWS suitable. Can be used to create the CDP VPC and Subnets. Note that this module is called from the terraform-cdp-aws-prereqs module. |
 | [terraform-aws-tgw](modules/terraform-aws-tgw/README.md) | Module for creation of AWS Transity Gateway (TGW) and attaching a specified list of VPCs via the TGW. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a fully private networking configuration where a CDP VPC and Networking VPC are connected using the Transit Gateway. |
 | [terraform-aws-proxy](modules/terraform-aws-proxy/README.md) | Module to create and configure and EC2 Auto-Scaling Group for a highly available Squid Proxy service with Network Load Balancer (NLB) to forward traffic to the proxy instances. This module can be used to assist in deploying Cloudera Data Platform (CDP) Public Cloud in a fully private networking configuration where a the CDP Environments uses a proxy config via the NLB. |

modules/terraform-aws-cred-permissions/.terraform-docs.yaml

@@ -0,0 +1,21 @@
+formatter: markdown
+header-from: doc_fragments/header.md
+settings:
+  anchor: true
+  color: true
+  default: true
+  escape: true
+  html: true
+  indent: 2
+  required: true
+  sensitive: true
+  type: true
+
+
+sort:
+  enabled: true
+  by: required
+
+output:
+  file: README.md
+  mode: replace

modules/terraform-aws-cred-permissions/README.md

@@ -0,0 +1,65 @@
+<!-- BEGIN_TF_DOCS -->
+# Terraform Module for CDP Credential Prerequisites on AWS
+
+This module contains resource files and example variable definition files for creation of the Cloudera Data Platform (CDP) Public Cloud Cross Account Credential pre-requisite on AWS.
+
+Support for using a pre-existing Cross Account Role is provided via the `existing_xaccount_role_name` input variable. When this is set no policy or role resources are created. Instead a lookup of the details of the existing role takes place and the role ARN is returned.
+
+## Usage
+
+The [examples](./examples) directory has the following examples for Cross Account Credentials on AWS:
+
+* `ex01-minimal-inputs` uses the minimum set of inputs for the module where the Cross Account policy and roles are to be created.
+
+* `ex02-existing-role` passes a pre-existing Cross Account role to the module. In this case no resources are created.
+
+In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.
+
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | ~>5.30 |
+| <a name="requirement_time"></a> [time](#requirement\_time) | 0.9.1 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | ~>5.30 |
+| <a name="provider_time"></a> [time](#provider\_time) | 0.9.1 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_iam_role.cdp_xaccount_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role_policy.cdp_xaccount_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource |
+| [time_sleep.iam_propagation](https://registry.terraform.io/providers/hashicorp/time/0.9.1/docs/resources/sleep) | resource |
+| [aws_iam_policy_document.cdp_xaccount_role_policy_doc](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_role.existing_xaccount_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_role) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_existing_xaccount_role_name"></a> [existing\_xaccount\_role\_name](#input\_existing\_xaccount\_role\_name) | Name of existing CDP Cross Account Role. If set then no policy or role resources are created. | `string` | `null` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | Tags applied to provised resources | `map(any)` | `null` | no |
+| <a name="input_xaccount_account_id"></a> [xaccount\_account\_id](#input\_xaccount\_account\_id) | Account ID of the cross account. Required if xaccount resources are to be created. | `string` | `null` | no |
+| <a name="input_xaccount_account_policy_doc"></a> [xaccount\_account\_policy\_doc](#input\_xaccount\_account\_policy\_doc) | Contents of cross acount policy document. Required if xaccount resources are to be created. | `string` | `null` | no |
+| <a name="input_xaccount_external_id"></a> [xaccount\_external\_id](#input\_xaccount\_external\_id) | External ID of the cross account. Required if xaccount resources are to be created. | `string` | `null` | no |
+| <a name="input_xaccount_policy_name"></a> [xaccount\_policy\_name](#input\_xaccount\_policy\_name) | Cross Account Policy name. Required if xaccount resources are to be created. | `string` | `null` | no |
+| <a name="input_xaccount_role_name"></a> [xaccount\_role\_name](#input\_xaccount\_role\_name) | Cross account Assume role Name. Required if xaccount resources are to be created. | `string` | `null` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_aws_xaccount_role_arn"></a> [aws\_xaccount\_role\_arn](#output\_aws\_xaccount\_role\_arn) | Cross Account role ARN |
+| <a name="output_aws_xaccount_role_name"></a> [aws\_xaccount\_role\_name](#output\_aws\_xaccount\_role\_name) | Cross Account role name |
+<!-- END_TF_DOCS -->

modules/terraform-aws-cred-permissions/data.tf

@@ -0,0 +1,20 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+data "aws_iam_role" "existing_xaccount_role" {
+
+  count = local.create_xaccount_resources ? 0 : 1
+
+  name = var.existing_xaccount_role_name
+}

modules/terraform-aws-cred-permissions/defaults.tf

@@ -0,0 +1,19 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+locals {
+  # ------- Determine if resources should be created -------
+  create_xaccount_resources = (var.existing_xaccount_role_name == null)
+
+}

modules/terraform-aws-cred-permissions/doc_fragments/header.md

@@ -0,0 +1,15 @@
+# Terraform Module for CDP Credential Prerequisites on AWS
+
+This module contains resource files and example variable definition files for creation of the Cloudera Data Platform (CDP) Public Cloud Cross Account Credential pre-requisite on AWS.
+
+Support for using a pre-existing Cross Account Role is provided via the `existing_xaccount_role_name` input variable. When this is set no policy or role resources are created. Instead a lookup of the details of the existing role takes place and the role ARN is returned.
+
+## Usage
+
+The [examples](./examples) directory has the following examples for Cross Account Credentials on AWS:
+
+* `ex01-minimal-inputs` uses the minimum set of inputs for the module where the Cross Account policy and roles are to be created.
+
+* `ex02-existing-role` passes a pre-existing Cross Account role to the module. In this case no resources are created.
+
+In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.

modules/terraform-aws-cred-permissions/examples/ex01-minimal-inputs/main.tf

@@ -0,0 +1,62 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 1.5.7"
+  required_providers {
+    cdp = {
+      source  = "cloudera/cdp"
+      version = "~> 0.6.1"
+    }
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~>5.30"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.aws_region
+}
+
+module "ex01_minimal_inputs" {
+  source = "../.."
+
+  tags = var.tags
+
+  # Using CDP TF Provider cred pre-reqs data source for values of xaccount account_id and external_id
+  xaccount_account_id  = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.account_id
+  xaccount_external_id = data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.external_id
+
+  xaccount_policy_name        = "${var.env_prefix}-xaccount-policy"
+  xaccount_account_policy_doc = base64decode(data.cdp_environments_aws_credential_prerequisites.cdp_prereqs.policy)
+
+  xaccount_role_name = "${var.env_prefix}-xaccount-role"
+
+}
+
+# Use the CDP Terraform Provider to find the xaccount account and external ids
+data "cdp_environments_aws_credential_prerequisites" "cdp_prereqs" {}
+
+# ------- Outputs -------
+output "xaccount_role_arn" {
+  value = module.ex01_minimal_inputs.aws_xaccount_role_arn
+
+  description = "The ARN of the created Cross Account Role"
+}
+output "xaccount_role_name" {
+  value = module.ex01_minimal_inputs.aws_xaccount_role_name
+
+  description = "The name of the created Cross Account Role"
+}

modules/terraform-aws-cred-permissions/examples/ex01-minimal-inputs/terraform.tfvars.sample

@@ -0,0 +1,27 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ------- Global settings -------
+env_prefix = "<ENTER_VALUE>" # Required name prefix for cloud and CDP resources, e.g. cldr1
+
+# ------- Cloud Settings -------
+aws_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eu-west-1
+
+# ------- Resource Tagging -------
+# **NOTE: An example of how to specify tags is below; uncomment & edit if required
+tags = {
+  owner   = "<ENTER_VALUE>"
+  project = "<ENTER_VALUE>"
+  enddate = "<ENTER_VALUE>"
+}

modules/terraform-aws-cred-permissions/examples/ex01-minimal-inputs/variables.tf

@@ -0,0 +1,31 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ------- Global settings -------
+variable "aws_region" {
+  type        = string
+  description = "Region which Cloud resources will be created"
+}
+
+variable "env_prefix" {
+  type        = string
+  description = "Shorthand name for the environment. Used in resource descriptions"
+}
+
+variable "tags" {
+  type        = map(any)
+  description = "Tags applied to provised resources"
+
+  default = null
+}

modules/terraform-aws-cred-permissions/examples/ex02-existing-role/main.tf

@@ -0,0 +1,50 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  required_version = ">= 1.5.7"
+  required_providers {
+    cdp = {
+      source  = "cloudera/cdp"
+      version = "~> 0.6.1"
+    }
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~>5.30"
+    }
+  }
+}
+
+provider "aws" {
+  region = var.aws_region
+}
+
+module "ex01_existing_role" {
+  source = "../.."
+
+  existing_xaccount_role_name = var.existing_xaccount_role_name
+
+}
+
+# ------- Outputs -------
+output "xaccount_role_arn" {
+  value = module.ex01_existing_role.aws_xaccount_role_arn
+
+  description = "The ARN of the created Cross Account Role"
+}
+output "xaccount_role_name" {
+  value = module.ex01_existing_role.aws_xaccount_role_name
+
+  description = "The name of the created Cross Account Role"
+}

modules/terraform-aws-cred-permissions/examples/ex02-existing-role/terraform.tfvars.sample

@@ -0,0 +1,19 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ------- Global settings -------
+existing_xaccount_role_name = "<ENTER_VALUE>" # Name of pre-existing cross account role, e.g. cldr1-xaccount-role
+
+# ------- Cloud Settings -------
+aws_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eu-west-1

modules/terraform-aws-cred-permissions/examples/ex02-existing-role/variables.tf

@@ -0,0 +1,24 @@
+# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# ------- Global settings -------
+variable "aws_region" {
+  type        = string
+  description = "Region which Cloud resources will be created"
+}
+
+variable "existing_xaccount_role_name" {
+  type        = string
+  description = "Name of pre-existing Cross Account Role"
+}