Juju provider not automatically handling SSH known_hosts when adding Juju machines via SSH.

[Vultaire]

Description

Unsure if this is a bug or a feature request, but: when experimenting with a Juju-on-LXD Terraform script, where I spin up LXDs and then add them to Juju via SSH, one problem which came up is prompting for verification of SSH host keys. If I have a Terraform script like the above which spins up LXDs and prepopulates their known_hosts files to allow SSH login, I would hope that I can then add them as Juju machines via SSH address.

What happens is: during "terraform plan", yes/no prompts pop up blocking deploy of machines. These prompts are also mixed with other Terraform output, so not only does it break automation, but it's not immediately clear that it is waiting for a "yes" response for each machine added in this way.

The local-exec provisioner can be used to work around this (see notes), but as that is discouraged in Terraform, it seemed worth filing this bug.

Urgency

Casually reporting

Terraform Juju Provider version

0.14.0

Terraform version

v1.9.6-dev (via snap, channel latest/stable)

Juju version

3.5.2

Terraform Configuration(s)

terraform {
  required_providers {
    juju = {
      source  = "juju/juju"
      version = "0.14.0"
    }
    lxd = {
      source = "terraform-lxd/lxd"
      version = "2.3.0"
    }
  }
}

provider "juju" {}

provider "lxd" {}

variable "ssh_public_key" {
  type = string
  default = "~/.ssh/id_rsa.pub"
}

variable "ssh_private_key" {
  type = string
  default = "~/.ssh/id_rsa"
}

resource "lxd_instance" "juju_machine_1" {
  name = "juju-machine-1"
  image = "ubuntu:jammy"
}

resource "lxd_instance_file" "juju_machine_1_keys" {
  instance = lxd_instance.juju_machine_1.name
  source_path = pathexpand(var.ssh_public_key)
  target_path = "/home/ubuntu/.ssh/authorized_keys"
  mode = "0600"
  # These are implicitly set, but then get prompted for replacement on updates
  # if not explicitly set.  Possible LXD provider bug.
  uid = 1000
  gid = 1000
}

resource "juju_model" "machine_test" {
  name = "machine-test"
}

resource "juju_machine" "juju_machine_1" {
  model = juju_model.machine_test.name
  ssh_address = "ubuntu@${lxd_instance.juju_machine_1.ipv4_address}"
  public_key_file = pathexpand(var.ssh_public_key)
  private_key_file = pathexpand(var.ssh_private_key)

  depends_on = [lxd_instance_file.juju_machine_1_keys]
}

resource "juju_application" "ubuntu" {
  name = "ubuntu"
  model = juju_model.machine_test.name
  charm {
    name = "ubuntu"
    channel = "latest/stable"
    base = "[email protected]"
  }
  units = 1
  placement = juju_machine.juju_machine_1.machine_id
}

Reproduce / Test

terraform init; terraform apply

Debug/Panic Output

Including only the relevant portion:

[...]
2024-09-18T08:27:00.124-0700 [TRACE] provider.terraform-provider-juju_v0.14.0: Calling provider defined Resource Create: @module=sdk.framework tf_req_id=18e41831-6a82-0042-dfd2-8837ee90a85a tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/[email protected]/internal/fwserver/server_createresource.go:100 tf_provider_addr=registry.terraform.io/juju/juju tf_resource_type=juju_machine timestamp=2024-09-18T08:27:00.124-0700
2024-09-18T08:27:00.124-0700 [TRACE] provider.terraform-provider-juju_v0.14.0: ModelUUID cache looking for "machine-test": @caller=github.com/juju/terraform-provider-juju/internal/juju/client.go:243 @module=juju.client machine-test="uuid(b20d4cf9-c128-4531-85b1-0ccec73bb38a) type(iaas)" timestamp=2024-09-18T08:27:00.124-0700
2024-09-18T08:27:00.124-0700 [TRACE] provider.terraform-provider-juju_v0.14.0: Found uuid for "machine-test" in cache: @caller=github.com/juju/terraform-provider-juju/internal/juju/client.go:243 @module=juju.client timestamp=2024-09-18T08:27:00.124-0700
The authenticity of host '10.97.53.11 (10.97.53.11)' can't be established.
ED25519 key fingerprint is SHA256:ABvGgO1SbE7207OAvmrrQbqDcpw3JqBl3AlAB95aMPY.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? 2024-09-18T08:27:03.797-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu (expand)" is waiting for "juju_machine.juju_machine_1"
2024-09-18T08:27:03.797-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu" is waiting for "juju_application.ubuntu (expand)"
2024-09-18T08:27:03.797-0700 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/juju/juju\"] (close)" is waiting for "juju_application.ubuntu"
2024-09-18T08:27:05.107-0700 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/juju/juju\"] (close)"
2024-09-18T08:27:08.800-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu" is waiting for "juju_application.ubuntu (expand)"
2024-09-18T08:27:08.800-0700 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/juju/juju\"] (close)" is waiting for "juju_application.ubuntu"
2024-09-18T08:27:08.800-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu (expand)" is waiting for "juju_machine.juju_machine_1"
2024-09-18T08:27:10.108-0700 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/juju/juju\"] (close)"
juju_machine.juju_machine_1: Still creating... [10s elapsed]
2024-09-18T08:27:13.803-0700 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/juju/juju\"] (close)" is waiting for "juju_application.ubuntu"
2024-09-18T08:27:13.803-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu (expand)" is waiting for "juju_machine.juju_machine_1"
2024-09-18T08:27:13.803-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu" is waiting for "juju_application.ubuntu (expand)"
2024-09-18T08:27:15.109-0700 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/juju/juju\"] (close)"
2024-09-18T08:27:18.806-0700 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/juju/juju\"] (close)" is waiting for "juju_application.ubuntu"
2024-09-18T08:27:18.807-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu" is waiting for "juju_application.ubuntu (expand)"
2024-09-18T08:27:18.807-0700 [TRACE] dag/walk: vertex "juju_application.ubuntu (expand)" is waiting for "juju_machine.juju_machine_1"
2024-09-18T08:27:20.110-0700 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/juju/juju\"] (close)"
juju_machine.juju_machine_1: Still creating... [20s elapsed]
[...repeats the last block of messages every 10 seconds...]

Notes & References

Workaround exists:

• Use a terraform_data resource with the local-exec provisioner to add host keys via ssh-keyscan
• Add a dependency on the above resource prior to adding the machine

Here is a concrete patch which implements the workaround for the provided example configuration:

diff --git a/sscce.tf b/sscce.tf
--- a/sscce.tf
+++ b/sscce.tf
@@ -45,13 +45,30 @@ resource "juju_model" "machine_test" {
   name = "machine-test"
 }
 
+resource "terraform_data" "accept_ssh_host_keys" {
+  # NOTE: Adding SSH machines like this is not fully automatic: there's an SSH host prompt.
+  # We need to finesse it with some glue.
+
+  # Drop previous host keys, if any
+  provisioner "local-exec" {
+    command = "ssh-keygen -f ${pathexpand("~/.ssh/known_hosts")} -R ${lxd_instance.juju_machine_1.ipv4_address}"
+  }
+
+  # Add new host keys to ~/.ssh/known_hosts
+  provisioner "local-exec" {
+    command = "ssh-keyscan ${lxd_instance.juju_machine_1.ipv4_address} >> ${pathexpand("~/.ssh/known_hosts")}"
+  }
+
+  depends_on = [lxd_instance_file.juju_machine_1_keys]
+}
+
 resource "juju_machine" "juju_machine_1" {
   model = juju_model.machine_test.name
   ssh_address = "ubuntu@${lxd_instance.juju_machine_1.ipv4_address}"
   public_key_file = pathexpand(var.ssh_public_key)
   private_key_file = pathexpand(var.ssh_private_key)
 
-  depends_on = [lxd_instance_file.juju_machine_1_keys]
+  depends_on = [lxd_instance_file.juju_machine_1_keys, terraform_data.accept_ssh_host_keys]
 }
 
 resource "juju_application" "ubuntu" {

[hmlanigan]

@Vultaire Have you tried the juju_ssh_keys resource? Or cloudinit_userdata in model config?