Terragrunt gives error but running debug terraform command completes successfully

[ggprod]

Describe the bug

I have a terragrunt configuration that deploys a terraform module I built for dataform (using the Google cloud beta provider). When I attempt to run terragrunt apply on the configuration it fails with the error:

DEBU[0009] Variables passed to terraform are located in "/Users/markpevec/Eshyft/infra/envs/data/dataform/terragrunt-debug.tfvars.json"  prefix=[/Users/markpevec/Eshyft/infra/envs/data/dataform] 
DEBU[0009] Run this command to replicate how terraform was invoked:  prefix=[/Users/markpevec/Eshyft/infra/envs/data/dataform] 
DEBU[0009]      terraform -chdir="/Users/markpevec/Eshyft/infra/envs/data/dataform/.terragrunt-cache/MeNHnUQtAFOc3wsY088_yVP6PAk/OwWZsN3z3TVN3gYjx26q-qeGwp4/modules/dataform" apply -var-file="/Users/markpevec/Eshyft/infra/envs/data/dataform/terragrunt-debug.tfvars.json"   prefix=[/Users/markpevec/Eshyft/infra/envs/data/dataform] 
DEBU[0010] Running command: terraform apply              prefix=[/Users/markpevec/Eshyft/infra/envs/data/dataform] 
╷
│ Error: Variables not allowed
│ 
│   on <value for var.repositories> line 1:
│   (source code not available)
│ 
│ Variables may not be used here.
╵
ERRO[0012] terraform invocation failed in /Users/markpevec/Eshyft/infra/envs/data/dataform/.terragrunt-cache/MeNHnUQtAFOc3wsY088_yVP6PAk/OwWZsN3z3TVN3gYjx26q-qeGwp4/modules/dataform  prefix=[/Users/markpevec/Eshyft/infra/envs/data/dataform] 
ERRO[0012] 1 error occurred:
        * [/Users/markpevec/Eshyft/infra/envs/data/dataform/.terragrunt-cache/MeNHnUQtAFOc3wsY088_yVP6PAk/OwWZsN3z3TVN3gYjx26q-qeGwp4/modules/dataform] exit status 1

When I run the debug command it completes/applies successfully:

markpevec@Marks-Mac-mini dataform % terraform -chdir="/Users/markpevec/Eshyft/infra/envs/data/dataform/.terragrunt-cache/MeNHnUQtAFOc3wsY088_yVP6PAk/OwWZsN3z3TVN3gYjx26q-qeGwp4/modules/dataform" apply -var-file="/Users/markpevec/Eshyft/infra/envs/data/dataform/terragrunt-debug.tfvars.json"

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_dataform_repository.repositories["main"] will be created
  + resource "google_dataform_repository" "repositories" {
      + display_name     = "Main Dataform repository"
      + effective_labels = (known after apply)
      + id               = (known after apply)
      + name             = "main"
      + project          = "eshyft-data"
      + region           = "us-east4"
      + service_account  = "[email protected]"
      + terraform_labels = (known after apply)

      + git_remote_settings {
          + default_branch = "dev"
          + token_status   = (known after apply)
          + url            = "[email protected]:leverage-it/dataform.git"

          + ssh_authentication_config {
              + host_public_key                 = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO"
              + user_private_key_secret_version = "projects/710596952899/secrets/dataform-repo-ssh-private-key/versions/1"
            }
        }

      + workspace_compilation_overrides {
          + default_database = "eshyft-data"
          + schema_suffix    = "${workspaceName}"
        }
    }

  # google_dataform_repository_release_config.releases["main:dev"] will be created
  + resource "google_dataform_repository_release_config" "releases" {
      + cron_schedule                    = "0 4 * * *"
      + git_commitish                    = "dev"
      + id                               = (known after apply)
      + name                             = "dev"
      + project                          = "eshyft-data"
      + recent_scheduled_release_records = (known after apply)
      + region                           = "us-east4"
      + repository                       = "main"

      + code_compilation_config {
          + assertion_schema = "dataform_dev"
          + default_database = "eshyft-data"
          + default_schema   = "dataform_dev"
        }
    }

  # google_dataform_repository_release_config.releases["main:prod"] will be created
  + resource "google_dataform_repository_release_config" "releases" {
      + cron_schedule                    = "0 4 * * *"
      + git_commitish                    = "main"
      + id                               = (known after apply)
      + name                             = "prod"
      + project                          = "eshyft-data"
      + recent_scheduled_release_records = (known after apply)
      + region                           = "us-east4"
      + repository                       = "main"

      + code_compilation_config {
          + assertion_schema = "dataform_prod"
          + default_database = "eshyft-data"
          + default_schema   = "dataform_prod"
        }
    }

  # google_dataform_repository_workflow_config.workflows["main:dev:main"] will be created
  + resource "google_dataform_repository_workflow_config" "workflows" {
      + cron_schedule                      = "0 5 * * *"
      + id                                 = (known after apply)
      + name                               = "dev_main"
      + project                            = "eshyft-data"
      + recent_scheduled_execution_records = (known after apply)
      + region                             = "us-east4"
      + release_config                     = (known after apply)
      + repository                         = "main"

      + invocation_config {
          + fully_refresh_incremental_tables_enabled = true
          + service_account                          = "[email protected]"
          + transitive_dependencies_included         = true
          + transitive_dependents_included           = true
        }
    }

  # google_dataform_repository_workflow_config.workflows["main:prod:main"] will be created
  + resource "google_dataform_repository_workflow_config" "workflows" {
      + cron_schedule                      = "0 5 * * *"
      + id                                 = (known after apply)
      + name                               = "prod_main"
      + project                            = "eshyft-data"
      + recent_scheduled_execution_records = (known after apply)
      + region                             = "us-east4"
      + release_config                     = (known after apply)
      + repository                         = "main"

      + invocation_config {
          + fully_refresh_incremental_tables_enabled = true
          + service_account                          = "[email protected]"
          + transitive_dependencies_included         = true
          + transitive_dependents_included           = true
        }
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + release_config_ids  = {
      + "main:dev"  = (known after apply)
      + "main:prod" = (known after apply)
    }
  + repository_ids      = {
      + main = (known after apply)
    }
  + workflow_config_ids = {
      + "main:dev:main"  = (known after apply)
      + "main:prod:main" = (known after apply)
    }

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_dataform_repository.repositories["main"]: Creating...
google_dataform_repository.repositories["main"]: Creation complete after 0s [id=projects/eshyft-data/locations/us-east4/repositories/main]
google_dataform_repository_release_config.releases["main:dev"]: Creating...
google_dataform_repository_release_config.releases["main:prod"]: Creating...
google_dataform_repository_release_config.releases["main:prod"]: Creation complete after 1s [id=projects/eshyft-data/locations/us-east4/repositories/main/releaseConfigs/prod]
google_dataform_repository_release_config.releases["main:dev"]: Creation complete after 1s [id=projects/eshyft-data/locations/us-east4/repositories/main/releaseConfigs/dev]
google_dataform_repository_workflow_config.workflows["main:dev:main"]: Creating...
google_dataform_repository_workflow_config.workflows["main:prod:main"]: Creating...
google_dataform_repository_workflow_config.workflows["main:dev:main"]: Creation complete after 0s [id=projects/eshyft-data/locations/us-east4/repositories/main/workflowConfigs/dev_main]
google_dataform_repository_workflow_config.workflows["main:prod:main"]: Creation complete after 0s [id=projects/eshyft-data/locations/us-east4/repositories/main/workflowConfigs/prod_main]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

release_config_ids = {
  "main:dev" = "projects/eshyft-data/locations/us-east4/repositories/main/releaseConfigs/dev"
  "main:prod" = "projects/eshyft-data/locations/us-east4/repositories/main/releaseConfigs/prod"
}
repository_ids = {
  "main" = "projects/eshyft-data/locations/us-east4/repositories/main"
}
workflow_config_ids = {
  "main:dev:main" = "projects/eshyft-data/locations/us-east4/repositories/main/workflowConfigs/dev_main"
  "main:prod:main" = "projects/eshyft-data/locations/us-east4/repositories/main/workflowConfigs/prod_main"
}

Steps To Reproduce

Run terragrunt in debug mode

terragrunt apply --terragrunt-log-level debug --terragrunt-debug

It errors, with output as shown above, then run the terraform debug command

terraform -chdir="/Users/markpevec/Eshyft/infra/envs/data/dataform/.terragrunt-cache/MeNHnUQtAFOc3wsY088_yVP6PAk/OwWZsN3z3TVN3gYjx26q-qeGwp4/modules/dataform" apply -var-file="/Users/markpevec/Eshyft/infra/envs/data/dataform/terragrunt-debug.tfvars.json"

and it completes successfully with the output show above

terragrunt.hcl:

terraform {
  source = "${get_parent_terragrunt_dir()}/..//modules/dataform"
}

include {
  path = find_in_parent_folders()
}

dependency "project" {
  config_path = "${get_parent_terragrunt_dir()}/data/project"
}

dependency "service_accounts" {
  config_path = "${get_parent_terragrunt_dir()}/data/service-accounts"
}

dependency "bigquery_datasets_dataform" {
  config_path = "${get_parent_terragrunt_dir()}/data/bigquery-datasets:dataform"
}

inputs = {
  project_id = dependency.project.outputs.project_id
  region = "us-east4"
  repositories = {
    main = {
      name = "main"
      display_name = "Main Dataform repository"
      service_account = dependency.service_accounts.outputs.service_accounts_map["dataform-dev"].email
      git_remote_settings = {
        url = "[email protected]:leverage-it/dataform.git"
        default_branch = "dev"
        ssh_authentication_config = {
          user_private_key_secret_version = "projects/710596952899/secrets/dataform-repo-ssh-private-key/versions/1"
          host_public_key = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO"
        }
      }
      workspace_compilation_overrides = {
        default_database = dependency.project.outputs.project_id
        schema_suffix = "$${workspaceName}"
      }
      release_configs = {
        dev = {
          git_commitish = "dev"
          cron_schedule = "0 4 * * *"
          code_compilation_config = {
            default_database = dependency.project.outputs.project_id
            default_schema = dependency.bigquery_datasets_dataform.outputs.datasets["dataform_dev"].dataset_id
            assertion_schema = dependency.bigquery_datasets_dataform.outputs.datasets["dataform_dev"].dataset_id
            workflow_configs = {
              main = {
                cron_schedule = "0 5 * * *"
                invocation_config = {
                  service_account = dependency.service_accounts.outputs.service_accounts_map["dataform-dev"].email
                  fully_refresh_incremental_tables_enabled = true
                  transitive_dependencies_included = true
                  transitive_dependents_included = true
                }
              }
            }
          }
        }

        prod = {
          git_commitish = "main"
          cron_schedule = "0 4 * * *"
          code_compilation_config = {
            default_database = dependency.project.outputs.project_id
            default_schema = dependency.bigquery_datasets_dataform.outputs.datasets["dataform_prod"].dataset_id
            assertion_schema = dependency.bigquery_datasets_dataform.outputs.datasets["dataform_prod"].dataset_id
            workflow_configs = {
              main = {
                cron_schedule = "0 5 * * *"
                invocation_config = {
                  service_account = dependency.service_accounts.outputs.service_accounts_map["dataform-prod"].email
                  fully_refresh_incremental_tables_enabled = true
                  transitive_dependencies_included = true
                  transitive_dependents_included = true
                }
              }
            }
          }
        }
      }
    }
  }
}

terraform module variables.tf:

variable "project_id" {
  description = "The GCP project ID"
}

variable "region" {
  description = "The region to create the Dataform repository in"
}

variable "repositories" {
  type = map(object({
    name = string
    display_name = optional(string)
    service_account = optional(string)
    labels = optional(map(string))
    npmrc_environment_variables_secret_version = optional(string)
    git_remote_settings = optional(object({
      url = string
      default_branch = string
      authentication_token_secret_version = optional(string)
      ssh_authentication_config = optional(object({
        user_private_key_secret_version = string
        host_public_key = string
      }))
    }))
    workspace_compilation_overrides = optional(object({
      default_database = optional(string)
      schema_suffix = optional(string)
      table_prefix = optional(string)
    }))
    iam_bindings = optional(map(list(string)))
    release_configs = optional(map(object({
      git_commitish = string
      cron_schedule = optional(string)
      time_zone = optional(string) # if not specified then default to UTC
      code_compilation_config = optional(object({
        default_database = optional(string)
        default_schema = optional(string)
        default_location = optional(string)
        assertion_schema = optional(string)
        database_suffix = optional(string)
        schema_suffix = optional(string)
        table_prefix = optional(string)
        vars = optional(map(string))
        workflow_configs = optional(map(object({
          cron_schedule = optional(string)
          time_zone = optional(string) # if not specified then default to UTC
          invocation_config = optional(object({
            included_targets = optional(map(object({
              database = optional(string)
              schema = optional(string)
              name = optional(string)
            })))
            included_tags = optional(list(string))
            transitive_dependencies_included = optional(bool)
            transitive_dependents_included = optional(bool)
            fully_refresh_incremental_tables_enabled = optional(bool)
            service_account = optional(string)
          }))
        })))
      }))
    })))
  }))
  description = "The repositories to create in Dataform"
  default = {}
}

terraform module main.tf:

locals {
  
  iam_bindings = merge(
    [for k1,v1 in var.repositories:
      {for k2,v2 in v1.iam_bindings:
        "${k1}:${k2}" => v2
      } if v1.iam_bindings != null
    ]...
  )
  release_configs = merge(
    [for k1,v1 in var.repositories: 
      {for k2,v2 in v1.release_configs:
        "${k1}:${k2}" => v2
      } if v1.release_configs != null
    ]...
  )
  workflow_configs = merge(
    [for k1,v1 in var.repositories: 
      merge(
        [for k2,v2 in v1.release_configs: 
          {for k3,v3 in v2.code_compilation_config.workflow_configs:
            "${k1}:${k2}:${k3}" => v3
          } if try(v2.code_compilation_config.workflow_configs, null) != null
        ]...
      ) if v1.release_configs != null
    ]...
  )
}

resource "google_dataform_repository" "repositories" {
  provider = google-beta
  for_each = var.repositories

  project = var.project_id
  region = var.region
  name = each.value.name
  display_name = each.value.display_name

  service_account = each.value.service_account
  
  npmrc_environment_variables_secret_version = each.value.npmrc_environment_variables_secret_version

  labels = each.value.labels

  dynamic "git_remote_settings" {
    for_each = each.value.git_remote_settings != null ? [each.value.git_remote_settings] : []
    content {
      url = git_remote_settings.value.url
      default_branch = git_remote_settings.value.default_branch
      authentication_token_secret_version = git_remote_settings.value.authentication_token_secret_version
      dynamic "ssh_authentication_config" {
        for_each = git_remote_settings.value.ssh_authentication_config != null ? [git_remote_settings.value.ssh_authentication_config] : []
        content {
          user_private_key_secret_version = ssh_authentication_config.value.user_private_key_secret_version
          host_public_key = ssh_authentication_config.value.host_public_key
        }
      }
    }
  }

  dynamic "workspace_compilation_overrides" {
    for_each = each.value.workspace_compilation_overrides != null ? [each.value.workspace_compilation_overrides] : []
    content {
      default_database = workspace_compilation_overrides.value.default_database
      schema_suffix = workspace_compilation_overrides.value.schema_suffix
      table_prefix = workspace_compilation_overrides.value.table_prefix
    }
  }
}

resource "google_dataform_repository_iam_binding" "bindings" {
  provider = google-beta
  for_each = local.iam_bindings

  project = var.project_id
  region = var.region
  repository = google_dataform_repository.repositories[split(":", each.key)[0]]["name"]
  role = split(":", each.key)[1]
  members = each.value
}

resource "google_dataform_repository_release_config" "releases" {
  provider = google-beta
  for_each = local.release_configs

  project = var.project_id
  region = var.region
  repository = google_dataform_repository.repositories[split(":", each.key)[0]]["name"]

  name          = split(":", each.key)[1]
  git_commitish = each.value.git_commitish
  cron_schedule = each.value.cron_schedule
  time_zone     = each.value.time_zone

  dynamic "code_compilation_config" {
    for_each = each.value.code_compilation_config != null ? [each.value.code_compilation_config] : []
    content {
      default_database = code_compilation_config.value.default_database
      default_schema   = code_compilation_config.value.default_schema
      default_location = code_compilation_config.value.default_location
      assertion_schema = code_compilation_config.value.assertion_schema
      database_suffix  = code_compilation_config.value.database_suffix
      schema_suffix    = code_compilation_config.value.schema_suffix
      table_prefix     = code_compilation_config.value.table_prefix
      vars = code_compilation_config.value.vars
    }
  }
}

resource "google_dataform_repository_workflow_config" "workflows" {
  provider = google-beta
  for_each = local.workflow_configs

  project = var.project_id
  region = var.region
  repository = google_dataform_repository.repositories[split(":", each.key)[0]]["name"]
  release_config = google_dataform_repository_release_config.releases["${split(":", each.key)[0]}:${split(":", each.key)[1]}"]["id"]
  name           = "${split(":", each.key)[1]}_${split(":", each.key)[2]}"

  cron_schedule   = each.value.cron_schedule
  time_zone       = each.value.time_zone

  dynamic "invocation_config" {
    for_each = each.value.invocation_config != null ? [each.value.invocation_config] : []
    content {
      dynamic "included_targets" {
        for_each = invocation_config.value.included_targets != null ? [invocation_config.value.included_targets] : []
        content {
          database = included_targets.value.database
          schema   = included_targets.value.schema
          name     = included_targets.value.name
        }
      }
      included_tags                            = invocation_config.value.included_tags
      transitive_dependencies_included         = invocation_config.value.transitive_dependencies_included
      transitive_dependents_included           = invocation_config.value.transitive_dependents_included
      fully_refresh_incremental_tables_enabled = invocation_config.value.fully_refresh_incremental_tables_enabled
      service_account                          = invocation_config.value.service_account
    }
  }
}

terraform module outputs.tf:

output "repository_ids" {
  value = {for k,v in google_dataform_repository.repositories: k => v.id}
}

output "release_config_ids" {
  value = {for k,v in google_dataform_repository_release_config.releases: k => v.id}
}

output "workflow_config_ids" {
  value = {for k,v in google_dataform_repository_workflow_config.workflows: k => v.id}
}

terraform module versions.tf:

terraform {
  required_version = ">= 1.8.4"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 5.34.0"
    }

    google-beta = {
      source  = "hashicorp/google-beta"
      version = ">= 5.34.0"
    }
  }
}

Expected behavior

Terragrunt should not be throwing the error as the terraform completes successfully and creates the intended resources

Nice to haves

• Terminal output
• Screenshots

Versions

• Terragrunt version: 0.58.9
• OpenTofu/Terraform version: TF 1.8.4
• Environment details (Ubuntu 20.04, Windows 10, etc.): Mac OS Monterey 12.7.5 on Mac mini (intel i7 chip). Google terraform providers (main and beta) 5.34.0

Additional context

Add any other context about the problem here.

[ggprod]

verified same problem with latest terragrunt 0.59.5 as well

[ggprod]

I wonder if this is being caused by my attempted escape in schema_suffix = "$${workspaceName}"

[ggprod]

yes, that was the issue, by changing that line to schema_suffix = "$$${workspaceName}" the problem disappears. I'm not sure if that is the intended escape sequence, if so it should be documented somewhere (I found it mentinoed in a github issue by searching on similar bugs)