Expose all parameters for CDP provider environment & datalake resources

modules/terraform-cdp-deploy/README.md

@@ -55,8 +55,10 @@ No resources.
 | <a name="input_aws_security_group_knox_id"></a> [aws\_security\_group\_knox\_id](#input\_aws\_security\_group\_knox\_id) | ID of the Knox Security Group for CDP environment. Required for CDP deployment on AWS. | `string` | `null` | no |
 | <a name="input_aws_vpc_id"></a> [aws\_vpc\_id](#input\_aws\_vpc\_id) | AWS Virtual Private Network ID. Required for CDP deployment on AWS. | `string` | `null` | no |
 | <a name="input_aws_xaccount_role_arn"></a> [aws\_xaccount\_role\_arn](#input\_aws\_xaccount\_role\_arn) | Cross Account Role ARN. Required for CDP deployment on AWS. | `string` | `null` | no |
+| <a name="input_azure_aks_private_dns_zone_id"></a> [azure\_aks\_private\_dns\_zone\_id](#input\_azure\_aks\_private\_dns\_zone\_id) | The ID of an existing private DNS zone used for the AKS. | `string` | `null` | no |
 | <a name="input_azure_cdp_gateway_subnet_names"></a> [azure\_cdp\_gateway\_subnet\_names](#input\_azure\_cdp\_gateway\_subnet\_names) | List of Azure Subnet Names CDP Endpoint Access Gateway. Required for CDP deployment on Azure. | `list(any)` | `null` | no |
 | <a name="input_azure_cdp_subnet_names"></a> [azure\_cdp\_subnet\_names](#input\_azure\_cdp\_subnet\_names) | List of Azure Subnet Names for CDP Resources. Required for CDP deployment on Azure. | `list(any)` | `null` | no |
+| <a name="input_azure_database_private_dns_zone_id"></a> [azure\_database\_private\_dns\_zone\_id](#input\_azure\_database\_private\_dns\_zone\_id) | The ID of an existing private DNS zone used for the database. | `string` | `null` | no |
 | <a name="input_azure_datalakeadmin_identity_id"></a> [azure\_datalakeadmin\_identity\_id](#input\_azure\_datalakeadmin\_identity\_id) | Datalake Admin Managed Identity ID. Required for CDP deployment on Azure. | `string` | `null` | no |
 | <a name="input_azure_idbroker_identity_id"></a> [azure\_idbroker\_identity\_id](#input\_azure\_idbroker\_identity\_id) | IDBroker Managed Identity ID. Required for CDP deployment on Azure. | `string` | `null` | no |
 | <a name="input_azure_log_identity_id"></a> [azure\_log\_identity\_id](#input\_azure\_log\_identity\_id) | Log Data Access Managed Identity ID. Required for CDP deployment on Azure. | `string` | `null` | no |
@@ -73,18 +75,32 @@ No resources.
 | <a name="input_cdp_admin_group_name"></a> [cdp\_admin\_group\_name](#input\_cdp\_admin\_group\_name) | Name of the CDP IAM Admin Group associated with the environment. Defaults to '<env\_prefix>-cdp-admin-group' if not specified. | `string` | `null` | no |
 | <a name="input_cdp_user_group_name"></a> [cdp\_user\_group\_name](#input\_cdp\_user\_group\_name) | Name of the CDP IAM User Group associated with the environment. Defaults to '<env\_prefix>-cdp-user-group' if not specified. | `string` | `null` | no |
 | <a name="input_cdp_xacccount_credential_name"></a> [cdp\_xacccount\_credential\_name](#input\_cdp\_xacccount\_credential\_name) | Name of the CDP Cross Account Credential. Defaults to '<env\_prefix>-xaccount-cred' if not specified. | `string` | `null` | no |
+| <a name="input_datalake_custom_instance_groups"></a> [datalake\_custom\_instance\_groups](#input\_datalake\_custom\_instance\_groups) | A set of custom instance groups for the datalake. Only applicable for CDP deployment on AWS. | <pre>list(<br>    object({<br>      name          = string,<br>      instance_type = optional(string)<br>    })<br>  )</pre> | `null` | no |
+| <a name="input_datalake_image"></a> [datalake\_image](#input\_datalake\_image) | The image to use for the datalake. Can only be used when the 'datalake\_version' parameter is set to null. You can use 'catalog' name and/or 'id' for selecting an image. | <pre>object({<br>    id      = optional(string)<br>    catalog = optional(string)<br>  })</pre> | `null` | no |
+| <a name="input_datalake_java_version"></a> [datalake\_java\_version](#input\_datalake\_java\_version) | The Java major version to use on the datalake cluster. | `number` | `null` | no |
 | <a name="input_datalake_name"></a> [datalake\_name](#input\_datalake\_name) | Name of the CDP datalake. Defaults to '<env\_prefix>-<aw\|az\|gc\|>-dl' if not specified. | `string` | `null` | no |
-| <a name="input_datalake_scale"></a> [datalake\_scale](#input\_datalake\_scale) | The scale of the datalake. Valid values are LIGHT\_DUTY, MEDIUM\_DUTY\_HA. | `string` | `null` | no |
-| <a name="input_datalake_version"></a> [datalake\_version](#input\_datalake\_version) | The Datalake Runtime version. Valid values are semantic versions, e.g. 7.2.16 | `string` | `"7.2.17"` | no |
+| <a name="input_datalake_recipes"></a> [datalake\_recipes](#input\_datalake\_recipes) | Additional recipes that will be attached on the datalake instances | <pre>list(<br>    object({<br>      instance_group_name = string,<br>      recipe_names        = string<br>    })<br>  )</pre> | `null` | no |
+| <a name="input_datalake_scale"></a> [datalake\_scale](#input\_datalake\_scale) | The scale of the datalake. Valid values are LIGHT\_DUTY, ENTERPRISE. | `string` | `null` | no |
+| <a name="input_datalake_version"></a> [datalake\_version](#input\_datalake\_version) | The Datalake Runtime version. Valid values are latest or a semantic version, e.g. 7.2.17 | `string` | `"latest"` | no |
 | <a name="input_enable_ccm_tunnel"></a> [enable\_ccm\_tunnel](#input\_enable\_ccm\_tunnel) | Flag to enable Cluster Connectivity Manager tunnel. If false then access from Cloud to CDP Control Plane CIDRs is required from via SG ingress | `bool` | `true` | no |
+| <a name="input_enable_outbound_load_balancer"></a> [enable\_outbound\_load\_balancer](#input\_enable\_outbound\_load\_balancer) | Create outbound load balancers for Azure environments. Only applicable for CDP deployment on Azure. | `bool` | `null` | no |
 | <a name="input_enable_raz"></a> [enable\_raz](#input\_enable\_raz) | Flag to enable Ranger Authorization Service (RAZ) | `bool` | `true` | no |
+| <a name="input_encryption_key_arn"></a> [encryption\_key\_arn](#input\_encryption\_key\_arn) | ARN of the AWS KMS CMK to use for the server-side encryption of AWS storage resources. Only applicable for CDP deployment on AWS. | `string` | `null` | no |
+| <a name="input_encryption_key_resource_group_name"></a> [encryption\_key\_resource\_group\_name](#input\_encryption\_key\_resource\_group\_name) | Name of the existing Azure resource group hosting the Azure Key Vault containing customer managed key which will be used to encrypt the Azure Managed Disk. Only applicable for CDP deployment on Azure. | `string` | `null` | no |
+| <a name="input_encryption_key_url"></a> [encryption\_key\_url](#input\_encryption\_key\_url) | URL of the key which will be used to encrypt the Azure Managed Disks. Only applicable for CDP deployment on Azure. | `string` | `null` | no |
 | <a name="input_endpoint_access_scheme"></a> [endpoint\_access\_scheme](#input\_endpoint\_access\_scheme) | The scheme for the workload endpoint gateway. PUBLIC creates an external endpoint that can be accessed over the Internet. PRIVATE which restricts the traffic to be internal to the VPC / Vnet. Relevant in Private Networks. | `string` | `null` | no |
 | <a name="input_env_prefix"></a> [env\_prefix](#input\_env\_prefix) | Shorthand name for the environment. Used in CDP resource descriptions. This will be used to construct the value of where any of the CDP resource variables (e.g. environment\_name, cdp\_iam\_admin\_group\_name) are not defined. | `string` | `null` | no |
 | <a name="input_environment_name"></a> [environment\_name](#input\_environment\_name) | Name of the CDP environment. Defaults to '<env\_prefix>-cdp-env' if not specified. | `string` | `null` | no |
+| <a name="input_freeipa_catalog"></a> [freeipa\_catalog](#input\_freeipa\_catalog) | Image catalog to use for FreeIPA image selection | `string` | `null` | no |
+| <a name="input_freeipa_image_id"></a> [freeipa\_image\_id](#input\_freeipa\_image\_id) | Image ID to use for creating FreeIPA instances | `string` | `null` | no |
+| <a name="input_freeipa_instance_type"></a> [freeipa\_instance\_type](#input\_freeipa\_instance\_type) | Instance Type to use for creating FreeIPA instances | `string` | `null` | no |
 | <a name="input_freeipa_instances"></a> [freeipa\_instances](#input\_freeipa\_instances) | The number of FreeIPA instances to create in the environment | `number` | `3` | no |
-| <a name="input_keypair_name"></a> [keypair\_name](#input\_keypair\_name) | SSH Keypair name in Cloud Service Provider. Required for CDP deployment on AWS. | `string` | `null` | no |
+| <a name="input_freeipa_recipes"></a> [freeipa\_recipes](#input\_freeipa\_recipes) | The recipes for the FreeIPA cluster | `set(string)` | `null` | no |
+| <a name="input_keypair_name"></a> [keypair\_name](#input\_keypair\_name) | SSH Keypair name in Cloud Service Provider. For CDP deployment on AWS, either 'keypair\_name' or 'public\_key\_text' needs to be set. | `string` | `null` | no |
 | <a name="input_multiaz"></a> [multiaz](#input\_multiaz) | Flag to specify that the FreeIPA and DataLake instances will be deployed across multi-availability zones. | `bool` | `true` | no |
-| <a name="input_public_key_text"></a> [public\_key\_text](#input\_public\_key\_text) | SSH Public key string for the nodes of the CDP environment. Required for CDP deployment on Azure. | `string` | `null` | no |
+| <a name="input_proxy_config_name"></a> [proxy\_config\_name](#input\_proxy\_config\_name) | Name of the proxy config to use for the environment. | `string` | `null` | no |
+| <a name="input_public_key_text"></a> [public\_key\_text](#input\_public\_key\_text) | SSH Public key string for the nodes of the CDP environment. Required for CDP deployment on Azure. For CDP deployment on AWS, either 'keypair\_name' or 'public\_key\_text' needs to be set. | `string` | `null` | no |
+| <a name="input_s3_guard_table_name"></a> [s3\_guard\_table\_name](#input\_s3\_guard\_table\_name) | Name for the DynamoDB table backing S3Guard. Only applicable for CDP deployment on AWS. | `string` | `null` | no |
 | <a name="input_use_public_ips"></a> [use\_public\_ips](#input\_use\_public\_ips) | Use public ip's for the CDP resources created within the Azure network. Required for CDP deployment on Azure. | `bool` | `null` | no |
 | <a name="input_use_single_resource_group"></a> [use\_single\_resource\_group](#input\_use\_single\_resource\_group) | Use a single resource group for all provisioned CDP resources. Required for CDP deployment on Azure. | `bool` | `true` | no |
 | <a name="input_workload_analytics"></a> [workload\_analytics](#input\_workload\_analytics) | Flag to specify if workload analytics should be enabled for the CDP environment | `bool` | `true` | no |

modules/terraform-cdp-deploy/defaults.tf

@@ -40,7 +40,7 @@ locals {
   datalake_scale = coalesce(
     var.datalake_scale,
     (var.deployment_template == "public" ?
-      "LIGHT_DUTY" : "MEDIUM_DUTY_HA"
+      "LIGHT_DUTY" : "ENTERPRISE"
     )
   )
 

modules/terraform-cdp-deploy/main.tf

@@ -46,7 +46,9 @@ module "cdp_on_aws" {
   public_subnet_ids = var.aws_public_subnet_ids
   # private_subnet_ids = var.aws_private_subnet_ids
   subnets_for_cdp = local.aws_subnets_for_cdp
+  # One of key settings below need to be set
   keypair_name    = var.keypair_name
+  public_key_text = var.public_key_text
 
   data_storage_location   = var.data_storage_location
   log_storage_location    = var.log_storage_location
@@ -59,6 +61,21 @@ module "cdp_on_aws" {
   idbroker_instance_profile_arn = var.aws_idbroker_instance_profile_arn
   log_instance_profile_arn      = var.aws_log_instance_profile_arn
 
+  # Optional parameters defaulting to null
+  freeipa_catalog       = var.freeipa_catalog
+  freeipa_image_id      = var.freeipa_image_id
+  freeipa_instance_type = var.freeipa_instance_type
+  freeipa_recipes       = var.freeipa_recipes
+
+  encryption_key_arn = var.encryption_key_arn
+
+  proxy_config_name   = var.proxy_config_name
+  s3_guard_table_name = var.s3_guard_table_name
+
+  datalake_custom_instance_groups = var.datalake_custom_instance_groups
+  datalake_image                  = var.datalake_image
+  datalake_java_version           = var.datalake_java_version
+  datalake_recipes                = var.datalake_recipes
 }
 
 # ------- Call sub-module for Azure Deployment -------
@@ -114,4 +131,24 @@ module "cdp_on_azure" {
   ranger_audit_identity_id  = var.azure_ranger_audit_identity_id
   log_identity_id           = var.azure_log_identity_id
   raz_identity_id           = var.azure_raz_identity_id
+
+  # Optional parameters defaulting to null
+  freeipa_catalog       = var.freeipa_catalog
+  freeipa_image_id      = var.freeipa_image_id
+  freeipa_instance_type = var.freeipa_instance_type
+  freeipa_recipes       = var.freeipa_recipes
+
+  enable_outbound_load_balancer = var.enable_outbound_load_balancer
+
+  encryption_key_resource_group_name = var.encryption_key_resource_group_name
+  encryption_key_url                 = var.encryption_key_url
+
+  azure_aks_private_dns_zone_id      = var.azure_aks_private_dns_zone_id
+  azure_database_private_dns_zone_id = var.azure_database_private_dns_zone_id
+
+  proxy_config_name = var.proxy_config_name
+
+  datalake_image        = var.datalake_image
+  datalake_java_version = var.datalake_java_version
+  datalake_recipes      = var.datalake_recipes
 }

modules/terraform-cdp-deploy/modules/aws/main.tf

@@ -37,6 +37,7 @@ resource "cdp_environments_aws_environment" "cdp_env" {
   }
 
   authentication = {
+    public_key    = var.public_key_text
     public_key_id = var.keypair_name
   }
 
@@ -48,10 +49,18 @@ resource "cdp_environments_aws_environment" "cdp_env" {
   freeipa = {
     instance_count_by_group = var.freeipa_instances
     multi_az                = var.multiaz
+    catalog                 = var.freeipa_catalog
+    image_id                = var.freeipa_image_id
+    instance_type           = var.freeipa_instance_type
+    recipes                 = var.freeipa_recipes
   }
 
-  workload_analytics = var.workload_analytics
-  enable_tunnel      = var.enable_ccm_tunnel
+  proxy_config_name   = var.proxy_config_name
+  s3_guard_table_name = var.s3_guard_table_name
+  workload_analytics  = var.workload_analytics
+  enable_tunnel       = var.enable_ccm_tunnel
+
+  encryption_key_arn = var.encryption_key_arn
   # tags               = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [
@@ -113,11 +122,15 @@ resource "cdp_datalake_aws_datalake" "cdp_datalake" {
   instance_profile        = var.idbroker_instance_profile_arn
   storage_bucket_location = var.data_storage_location
 
-  runtime           = var.datalake_version
+  runtime           = var.datalake_version == "latest" ? null : var.datalake_version
   scale             = var.datalake_scale
   enable_ranger_raz = var.enable_raz
   multi_az          = var.multiaz
 
+  custom_instance_groups = var.datalake_custom_instance_groups
+  image                  = var.datalake_image
+  java_version           = var.datalake_java_version
+  recipes                = var.datalake_recipes
   # tags = var.tags # NOTE: Waiting on provider fix
 
   depends_on = [

modules/terraform-cdp-deploy/modules/aws/variables.tf

@@ -86,37 +86,121 @@ variable "freeipa_instances" {
 
 }
 
+variable "freeipa_catalog" {
+  type = string
+
+  description = "Image catalog to use for FreeIPA image selection"
+
+}
+
+variable "freeipa_image_id" {
+  type = string
+
+  description = "Image ID to use for creating FreeIPA instances"
+
+}
+
+variable "freeipa_instance_type" {
+  type = string
+
+  description = "Instance Type to use for creating FreeIPA instances"
+
+}
+
+variable "freeipa_recipes" {
+  type = set(string)
+
+  description = "The recipes for the FreeIPA cluster"
+
+}
+
+variable "proxy_config_name" {
+  type = string
+
+  description = "Name of the proxy config to use for the environment."
+
+}
+
+variable "s3_guard_table_name" {
+  type = string
+
+  description = "Name for the DynamoDB table backing S3Guard. Only applicable for CDP deployment on AWS."
+
+}
+
 variable "workload_analytics" {
   type = bool
 
   description = "Flag to specify if workload analytics should be enabled for the CDP environment"
 
 }
 
+
 variable "datalake_scale" {
   type = string
 
-  description = "The scale of the datalake. Valid values are LIGHT_DUTY, MEDIUM_DUTY_HA."
+  description = "The scale of the datalake. Valid values are LIGHT_DUTY, ENTERPRISE."
 
   validation {
-    condition     = contains(["LIGHT_DUTY", "MEDIUM_DUTY_HA"], var.datalake_scale)
-    error_message = "Valid values for var: datalake_scale are (LIGHT_DUTY, MEDIUM_DUTY_HA)."
+    condition     = contains(["LIGHT_DUTY", "ENTERPRISE", "MEDIUM_DUTY_HA"], var.datalake_scale)
+    error_message = "Valid values for var: datalake_scale are (LIGHT_DUTY, ENTERPRISE, MEDIUM_DUTY_HA)."
   }
 
 }
 
 variable "datalake_version" {
   type = string
 
-  description = "The Datalake Runtime version. Valid values are semantic versions, e.g. 7.2.16"
+  description = "The Datalake Runtime version. Valid values are latest or a semantic version, e.g. 7.2.17"
 
   validation {
-    condition     = length(regexall("\\d+\\.\\d+.\\d+", var.datalake_version)) > 0
-    error_message = "Valid values for var: datalake_version must match semantic versioning conventions."
+    condition     = (var.datalake_version == "latest" ? true : length(regexall("\\d+\\.\\d+.\\d+", var.datalake_version)) > 0)
+    error_message = "Valid values for var: datalake_version are 'latest' or a semantic versioning conventions."
   }
 
+  default = "latest"
+}
+
+variable "datalake_custom_instance_groups" {
+  type = list(
+    object({
+      name          = string,
+      instance_type = optional(string)
+    })
+  )
+
+  description = "A set of custom instance groups for the datalake."
+
 }
 
+variable "datalake_image" {
+  type = object({
+    id      = optional(string)
+    catalog = optional(string)
+  })
+
+  description = "The image to use for the datalake. Can only be used when the 'datalake_version' parameter is set to null. You can use 'catalog' name and/or 'id' for selecting an image."
+
+}
+
+variable "datalake_java_version" {
+  type = number
+
+  description = "The Java major version to use on the datalake cluster."
+
+}
+
+variable "datalake_recipes" {
+  type = list(
+    object({
+      instance_group_name = string,
+      recipe_names        = string
+    })
+  )
+
+  description = "Additional recipes that will be attached on the datalake instances"
+
+}
 # ------- Cloud Service Provider Settings -------
 variable "region" {
   type        = string
@@ -127,7 +211,15 @@ variable "region" {
 variable "keypair_name" {
   type = string
 
-  description = "SSH Keypair name in Cloud Service Provider. Required for CDP deployment on AWS."
+  description = "SSH Keypair name in Cloud Service Provider. Either 'keypair_name' or 'public_key_text' needs to be set."
+
+  default = null
+}
+
+variable "public_key_text" {
+  type = string
+
+  description = "SSH Public key string for the nodes of the CDP environment. Either 'keypair_name' or 'public_key_text' needs to be set."
 
   default = null
 }
@@ -183,6 +275,14 @@ variable "endpoint_access_scheme" {
   }
 }
 
+variable "encryption_key_arn" {
+  type = string
+
+  description = "ARN of the AWS KMS CMK to use for the server-side encryption of AWS storage resources."
+
+}
+
+
 variable "data_storage_location" {
   type        = string
   description = "Data storage location."