Feature to support provisioning of Azure Files NFS for CML at runtime (…

…#39)

Signed-off-by: tsharma <[email protected]>

modules/terraform-azure-nfs/.terraform-docs.yaml

@@ -0,0 +1,21 @@
+formatter: markdown
+header-from: doc_fragments/header.md
+settings:
+  anchor: true
+  color: true
+  default: true
+  escape: true
+  html: true
+  indent: 2
+  required: true
+  sensitive: true
+  type: true
+
+
+sort:
+  enabled: true
+  by: required
+
+output:
+  file: README.md
+  mode: replace

modules/terraform-azure-nfs/README.md

@@ -0,0 +1,87 @@
+<!-- BEGIN_TF_DOCS -->
+# Terraform Module for Azure NFS
+
+This module contains resource files and example variable definition files for creation of the Azure NFS File Share required for Cloudera Machine Learning (CML) Public Cloud.
+
+* Provisions a storage account with Premium Tier and Disabled Https traffic only.
+* Creates a NFS file share of 100 GB in the storage account
+* Creates a private dns zone of type privatelink.file.core.windows.net
+* Creates a VNET link between CDP workload VNET and private DNS zone
+* Creates a private endpoint for NFS Storage Account (File sub-resource) for the specified subnets in the CDP VNET.
+* Creates a public IP , security group allowing port 22 from everywhere
+* Creates a ubuntu VM with public IP, security group in the CDP Subnets to which private endpoint was created.
+
+## Usage
+
+The [examples](./examples) directory has example Azure Cloud Service Provider deployments for different scenarios:
+
+* `ex01-nfs_fileshare` uses a set of inputs for the module.
+
+In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.
+
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
+| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | 3.45.0 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | 3.77.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [azurerm_linux_virtual_machine.nfs_vm](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/linux_virtual_machine) | resource |
+| [azurerm_network_interface.nfsvm_nic](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/network_interface) | resource |
+| [azurerm_network_interface_security_group_association.nfsvm_nic_sg](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/network_interface_security_group_association) | resource |
+| [azurerm_network_security_group.nfsvm_sg](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/network_security_group) | resource |
+| [azurerm_network_security_rule.nfsvm_sg_rule](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/network_security_rule) | resource |
+| [azurerm_private_dns_zone.nfs_privatednszone](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/private_dns_zone) | resource |
+| [azurerm_private_dns_zone_virtual_network_link.nfs_vnet_link](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/private_dns_zone_virtual_network_link) | resource |
+| [azurerm_private_endpoint.nfs_private_endpoint](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/private_endpoint) | resource |
+| [azurerm_public_ip.nfsvm_public_ip](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/public_ip) | resource |
+| [azurerm_storage_account.nfs_storage_account](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/storage_account) | resource |
+| [azurerm_storage_share.nfs_storage_share](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/resources/storage_share) | resource |
+| [azurerm_subnet.nfs_subnets](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/data-sources/subnet) | data source |
+| [azurerm_virtual_network.nfs_vnet](https://registry.terraform.io/providers/hashicorp/azurerm/3.45.0/docs/data-sources/virtual_network) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_azure_region"></a> [azure\_region](#input\_azure\_region) | Region for CDP | `string` | n/a | yes |
+| <a name="input_nfs_file_share_name"></a> [nfs\_file\_share\_name](#input\_nfs\_file\_share\_name) | nfs file share name | `string` | n/a | yes |
+| <a name="input_nfs_private_endpoint_target_subnet_names"></a> [nfs\_private\_endpoint\_target\_subnet\_names](#input\_nfs\_private\_endpoint\_target\_subnet\_names) | Subnet to which private endpoints are created | `list(string)` | n/a | yes |
+| <a name="input_nfs_storage_account_name"></a> [nfs\_storage\_account\_name](#input\_nfs\_storage\_account\_name) | NFS Storage account name | `string` | n/a | yes |
+| <a name="input_nfs_vnet_link_name"></a> [nfs\_vnet\_link\_name](#input\_nfs\_vnet\_link\_name) | Name for NFS VNET Link | `string` | n/a | yes |
+| <a name="input_private_endpoint_prefix"></a> [private\_endpoint\_prefix](#input\_private\_endpoint\_prefix) | Shorthand name for the environment. Used in resource descriptions | `string` | n/a | yes |
+| <a name="input_resourcegroup_name"></a> [resourcegroup\_name](#input\_resourcegroup\_name) | Resource Group Name | `string` | n/a | yes |
+| <a name="input_vnet_name"></a> [vnet\_name](#input\_vnet\_name) | Vnet name | `string` | n/a | yes |
+| <a name="input_create_vm_mounting_nfs"></a> [create\_vm\_mounting\_nfs](#input\_create\_vm\_mounting\_nfs) | Whether to create a VM which mounts this NFS | `bool` | `true` | no |
+| <a name="input_nfs_file_share_size"></a> [nfs\_file\_share\_size](#input\_nfs\_file\_share\_size) | NFS File Share size | `number` | `100` | no |
+| <a name="input_nfsvm_name"></a> [nfsvm\_name](#input\_nfsvm\_name) | Name for NFS VM | `string` | `null` | no |
+| <a name="input_nfsvm_nic_name"></a> [nfsvm\_nic\_name](#input\_nfsvm\_nic\_name) | Name for NFS VM NIC | `string` | `null` | no |
+| <a name="input_nfsvm_public_ip_name"></a> [nfsvm\_public\_ip\_name](#input\_nfsvm\_public\_ip\_name) | Name for NFS VM Public IP | `string` | `null` | no |
+| <a name="input_nfsvm_sg_name"></a> [nfsvm\_sg\_name](#input\_nfsvm\_sg\_name) | Name for NFS VM Security Group | `string` | `null` | no |
+| <a name="input_public_key_text"></a> [public\_key\_text](#input\_public\_key\_text) | SSH Public key string for the nodes of the CDP environment | `string` | `null` | no |
+| <a name="input_source_address_prefixes"></a> [source\_address\_prefixes](#input\_source\_address\_prefixes) | Source address prefixes for VM ssh access | `list(string)` | `null` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_nfs_file_share_url"></a> [nfs\_file\_share\_url](#output\_nfs\_file\_share\_url) | NFS File Share url |
+| <a name="output_nfs_storage_account_name"></a> [nfs\_storage\_account\_name](#output\_nfs\_storage\_account\_name) | NFS Storage Account Name |
+| <a name="output_nfs_vm_mount_path"></a> [nfs\_vm\_mount\_path](#output\_nfs\_vm\_mount\_path) | Path where NFS is mounted on the VM |
+| <a name="output_nfs_vm_public_ip"></a> [nfs\_vm\_public\_ip](#output\_nfs\_vm\_public\_ip) | NFS VM public IP address |
+| <a name="output_nfs_vm_username"></a> [nfs\_vm\_username](#output\_nfs\_vm\_username) | NFS VM Admin Username |
+<!-- END_TF_DOCS -->

modules/terraform-azure-nfs/data.tf

@@ -0,0 +1,27 @@
+# Copyright 2023 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+data "azurerm_subnet" "nfs_subnets" {
+
+  for_each = toset(var.nfs_private_endpoint_target_subnet_names)
+
+  name                 = each.value
+  virtual_network_name = var.vnet_name
+  resource_group_name  = var.resourcegroup_name
+}
+
+
+data "azurerm_virtual_network" "nfs_vnet" {
+  name                = var.vnet_name
+  resource_group_name = var.resourcegroup_name
+}

modules/terraform-azure-nfs/defaults.tf

@@ -0,0 +1,17 @@
+# Copyright 2023 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+locals {
+
+}

modules/terraform-azure-nfs/doc_fragments/header.md

@@ -0,0 +1,19 @@
+# Terraform Module for Azure NFS
+
+This module contains resource files and example variable definition files for creation of the Azure NFS File Share required for Cloudera Machine Learning (CML) Public Cloud.
+
+* Provisions a storage account with Premium Tier and Disabled Https traffic only.
+* Creates a NFS file share of 100 GB in the storage account
+* Creates a private dns zone of type privatelink.file.core.windows.net
+* Creates a VNET link between CDP workload VNET and private DNS zone
+* Creates a private endpoint for NFS Storage Account (File sub-resource) for the specified subnets in the CDP VNET.
+* Creates a public IP , security group allowing port 22 from everywhere
+* Creates a ubuntu VM with public IP, security group in the CDP Subnets to which private endpoint was created.
+
+## Usage
+
+The [examples](./examples) directory has example Azure Cloud Service Provider deployments for different scenarios:
+
+* `ex01-nfs_fileshare` uses a set of inputs for the module.
+
+In each directory an example `terraform.tfvars.sample` values file is included to show input variable values.

modules/terraform-azure-nfs/examples/ex01-nfs_fileshare/main.tf

@@ -0,0 +1,64 @@
+# Copyright 2023 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+provider "azurerm" {
+  features {
+    resource_group {
+      prevent_deletion_if_contains_resources = false
+    }
+  }
+}
+
+module "ex01_nfs_fileshare" {
+  source = "../.."
+
+  resourcegroup_name                       = var.resourcegroup_name
+  azure_region                             = var.azure_region
+  nfs_file_share_name                      = var.nfs_file_share_name
+  nfs_file_share_size                      = var.nfs_file_share_size
+  nfs_private_endpoint_target_subnet_names = var.nfs_private_endpoint_target_subnet_names
+  vnet_name                                = var.vnet_name
+  nfs_storage_account_name                 = var.nfs_storage_account_name
+  source_address_prefixes                  = var.source_address_prefixes
+  nfsvm_nic_name                           = var.nfsvm_nic_name
+  nfsvm_public_ip_name                     = var.nfsvm_public_ip_name
+  nfsvm_sg_name                            = var.nfsvm_sg_name
+  nfs_vnet_link_name                       = var.nfs_vnet_link_name
+  nfsvm_name                               = var.nfsvm_name
+  public_key_text                          = var.public_key_text
+  private_endpoint_prefix                  = var.private_endpoint_prefix
+}
+
+output "nfs_file_share_url" {
+  value = module.ex01_nfs_fileshare.nfs_file_share_url
+}
+output "nfs_vm_public_ip" {
+  value = module.ex01_nfs_fileshare.nfs_vm_public_ip
+}
+
+
+output "nfs_vm_username" {
+  value       = module.ex01_nfs_fileshare.nfs_vm_username
+  description = "NFS VM Admin Username"
+}
+
+output "nfs_vm_mount_path" {
+  value       = module.ex01_nfs_fileshare.nfs_vm_mount_path
+  description = "Path where NFS is mounted on the VM"
+}
+
+output "nfs_storage_account_name" {
+  value       = module.ex01_nfs_fileshare.nfs_storage_account_name
+  description = "NFS Storage Account Name"
+}

modules/terraform-azure-nfs/examples/ex01-nfs_fileshare/terraform.tfvars.sample

@@ -0,0 +1,32 @@
+# Copyright 2023 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+private_endpoint_prefix = "<ENTER_VALUE>"
+
+# ------- Cloud Settings -------
+azure_region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eastus
+
+resourcegroup_name                      = "<ENTER_VALUE>"
+nfs_file_share_name                     = "<ENTER_VALUE>"
+nfs_file_share_size                     = <ENTER_VALUE>
+nfs_private_endpoint_target_subnet_names = ["<ENTER_VALUE>", "<ENTER_VALUE>"]
+vnet_name                               = "<ENTER_VALUE>"
+nfs_storage_account_name                = "<ENTER_VALUE>"
+nfsvm_public_ip_name                    = "<ENTER_VALUE>"
+nfsvm_nic_name                          = "<ENTER_VALUE>"
+nfsvm_sg_name                           = <ENTER_VALUE>
+nfs_vnet_link_name                      = "<ENTER_VALUE>"
+nfsvm_name                              = "<ENTER_VALUE>"
+public_key_text                         = "<ENTER_VALUE>"
+source_address_prefixes                 = ["<ENTER_VALUE>", "<ENTER_VALUE>"]

modules/terraform-azure-nfs/examples/ex01-nfs_fileshare/variables.tf

@@ -0,0 +1,96 @@
+# Copyright 2023 Cloudera, Inc. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+variable "resourcegroup_name" {
+  type        = string
+  description = "Resource Group Name"
+}
+
+variable "vnet_name" {
+  type        = string
+  description = "Vnet name"
+
+}
+
+
+variable "azure_region" {
+  type        = string
+  description = "Region for CDP"
+
+}
+
+variable "private_endpoint_prefix" {
+  type        = string
+  description = "Shorthand name for the environment. Used in resource descriptions"
+}
+
+
+variable "nfs_file_share_name" {
+  type        = string
+  description = "nfs file share name"
+}
+
+variable "nfs_private_endpoint_target_subnet_names" {
+  type        = list(string)
+  description = "Subnets to which private endpoints are created"
+}
+
+variable "nfs_storage_account_name" {
+  type        = string
+  description = "NFS Storage account name"
+}
+
+variable "nfs_file_share_size" {
+  type        = number
+  description = "NFS File Share size"
+}
+
+
+variable "nfsvm_public_ip_name" {
+  type        = string
+  description = "Name for NFS VM Public IP"
+}
+
+variable "nfsvm_nic_name" {
+  type        = string
+  description = "Name for NFS VM NIC"
+}
+
+variable "nfsvm_sg_name" {
+  type        = string
+  description = "Name for NFS VM Security Group"
+}
+
+variable "nfs_vnet_link_name" {
+  type        = string
+  description = "Name for NFS VNET Link"
+}
+
+
+variable "nfsvm_name" {
+  type        = string
+  description = "Name for NFS VM"
+}
+
+variable "public_key_text" {
+  type = string
+
+  description = "SSH Public key string for the nodes of the CDP environment"
+}
+
+variable "source_address_prefixes" {
+  type        = list(string)
+  description = "Source address prefixes for VM ssh access"
+}

modules/terraform-azure-nfs/files/mount_nfs_on_vm.sh.tpl

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+apt-get -y update
+apt-get -y install nfs-common
+mkdir -p /mount/${nfs_storage_account_name}/${nfs_file_share_name}
+mount -t nfs ${nfs_storage_account_name}.file.core.windows.net:/${nfs_storage_account_name}/${nfs_file_share_name} /mount/${nfs_storage_account_name}/${nfs_file_share_name} -o vers=4,minorversion=1,sec=sys