Add additional Linux distros and Consul editions to Enos scenarios #25983

rebwill · 2024-03-15T18:47:15Z

This PR:

Adds Enos module support for additional Linux distros in multiple versions: Amazon Linux (amzn2), OpenSUSE Leap (15.5, 15.4), SUSE SLES (v15 SP5), as well as support for installing and using various packages with each one
Adds additional Linux distros into all scenarios that include distros in the matrix
Adds support for Consul enterprise edition
Adds consul_edition into all relevant scenarios
Updates CI scenario sampling to include new matrix items
Upgrades our RHEL versions from 8.8 and 9.1 to 8.9 and 9.3

This more than doubles the number of Linux distros/versions we test Vault on, and allows us to test integrations of Vault with Consul enterprise in addition to Consul community edition.

github-actions · 2024-03-15T19:01:07Z

CI Results:
All Go tests succeeded! ✅

github-actions · 2024-03-15T19:03:08Z

Build Results:
All builds succeeded! ✅

enos/enos-scenario-proxy.hcl

enos/modules/ec2_info/main.tf

enos/modules/vault_artifactory_artifact/locals.tf

enos/modules/vault_verify_raft_auto_join_voter/scripts/verify-raft-auto-join-voter.sh

jasonodonnell

Minor questions/nits, but looks good AFAICT!

ryancragun

What a journey!

I'm very excited to see this land. I've interspersed some thoughts along with feedback.

Before merge:

Fix sampling for zip bundles. Only test on Amazon linux and Ubuntu.
Remove the Consul Enterprise filtering out of CE samples
Remove the leap exclusions from the sample since they're excluded already by the scenario
Remove the host info module that we're not using
Address feedback for install-packages.sh
Use install_packages module in the vault_cluster module
An enterprise counterpart of the PR with enterprise samples. For that we'd just create a branch when this is ready to go against enterprise and add the samples as separate commit. That will allow us to test all of these changes there before we merge it here. When it comes time to merge you could rebase away the commits that aren't samples and then merge it when it CE change gets to enterprise.

Some followup tasks which can be done on this or later:

Move consul_version to the sample attributes and out of scenario matrices
Update the distro-repo-setup script to take in the DISTRO as well
Update the RHEL versions we test
Add Ubuntu 24.04 to the distros for Ubuntu

ryancragun · 2024-03-25T20:20:44Z

.github/workflows/test-run-enos-scenario-matrix.yml

@@ -97,6 +97,11 @@ jobs:
      ENOS_VAR_vault_product_version: ${{ needs.metadata.outputs.vault-version }}
      ENOS_VAR_vault_revision: ${{ inputs.vault-revision }}
      ENOS_VAR_vault_license_path: ./support/vault.hclic
+      ENOS_VAR_distro_version_amzn2: ${{ matrix.attributes.distro_version_amzn2 }}


We should move the consul_version into the sample attributes. Definitely not required in this PR but something we should track and do after this is merged. It'll reduce our matrix size but I think the entire sample field with attributes will become the new source of truth and we should make the enos CLI capable of reporting that.

Will create a ticket for this.

ryancragun · 2024-03-25T20:22:14Z

.github/workflows/test-run-enos-scenario-matrix.yml

+      ENOS_VAR_distro_version_leap: ${{ matrix.attributes.distro_version_leap }}
+      ENOS_VAR_distro_version_rhel: ${{ matrix.attributes.distro_version_rhel }}
+      ENOS_VAR_distro_version_sles: ${{ matrix.attributes.distro_version_sles }}
+      ENOS_VAR_distro_version_ubuntu: ${{ matrix.attributes.distro_version_ubuntu }}


Quite excited for additional RHEL and Ubuntu versions in the sample field! Related: Ubuntu 24.04 drops next month. I wonder if there are already AMIs out that we can add into our module so we can get a head start on testing for it?

I will opt not to include this in this particular PR, but will create a ticket for it.

ryancragun · 2024-03-25T20:25:52Z

enos/enos-globals.hcl

+  artifact_sources     = ["local", "crt", "artifactory"]
+  artifact_types       = ["bundle", "package"]
+  backends             = ["consul", "raft"]
+  backend_license_path = abspath(var.backend_license_path != null ? var.backend_license_path : joinpath(path.root, "./support/consul.hclic"))


I wrote it but couldn't remember if joinpath() requires the file to actually exist or not, the latter of which I didn't want to be the case. But I've looked it up and it just creates the path but doesn't stat it.

ryancragun · 2024-03-25T20:26:35Z

enos/enos-globals.hcl

+    # When installing Vault RPM packages, SLES searches for openssl by a different name
+    # than the one that comes pre-installed on the AMI. Therefore we add the
+    # "correctly" named one in our package installation before installing Vault.
+    # sles   = ["ncat", "openssl"]


Suggested change

# sles = ["ncat", "openssl"]

ryancragun · 2024-03-25T20:27:22Z

enos/enos-globals.hcl

+    amzn2 = ["nc"]
+    leap  = ["netcat", "openssl"]
+    rhel  = ["nc"]
+    # When installing Vault RPM packages, SLES searches for openssl by a different name


This is such a bummer. The other package does provide openssl but isn't named openssl, which RPM doesn't know how to handle.

enos/modules/vault_cluster/main.tf

ryancragun · 2024-03-25T21:05:48Z

enos/modules/vault_cluster/scripts/install-packages.sh

@@ -0,0 +1,45 @@
+#!/bin/bash


As they do the same thing, we probably ought to remove this script and use the install_packages module in vault_cluster instead.

enos/modules/vault_cluster/scripts/start-audit-socket-listener.sh

ryancragun · 2024-03-25T21:07:46Z

enos/modules/vault_cluster/variables.tf

@@ -1,6 +1,12 @@
 # Copyright (c) HashiCorp, Inc.
 # SPDX-License-Identifier: BUSL-1.1

+variable "arch" {


I don't see where we use the arch, consul_bind_addr, distro or package_manager variables in the module.

ryancragun · 2024-03-25T21:12:23Z

enos/enos-samples-ce-build.hcl

@@ -200,6 +217,7 @@ sample "build_ce_linux_amd64_zip" {
      arch            = ["amd64"]
      artifact_type   = ["bundle"]
      artifact_source = ["crt"]
+      consul_edition  = ["ce"]


For zip bundles I think we ought to only test on ubuntu and amz2 while leaving all of the distros for the system packages. The idea is that when we're testing our zip bundles we'll favor testing on distros that don't have vendor markup.

rebwill · 2024-04-05T19:16:20Z

Follow up tickets:
https://hashicorp.atlassian.net/browse/QT-698
https://hashicorp.atlassian.net/browse/QT-699
https://hashicorp.atlassian.net/browse/QT-700
https://hashicorp.atlassian.net/browse/QT-701
https://hashicorp.atlassian.net/browse/QT-702

ryancragun

Just a few tiny nits but otherwise I think we should get an Ent PR created to test with.

The only thing I'm concerned about is the install location on RHEL. Did that change on purpose/how did it work before?

enos/enos-globals.hcl

ryancragun · 2024-04-11T20:32:36Z

enos/enos-samples-ce-build.hcl

@@ -191,6 +191,7 @@ sample "build_ce_linux_amd64_zip" {
      arch            = ["amd64"]
      artifact_type   = ["bundle"]
      artifact_source = ["crt"]
+      distro          = ["amzn2", "ubuntu"]


ryancragun · 2024-04-11T20:35:36Z

enos/modules/ec2_info/main.tf

+# and accept SUSE's terms of use. You can do this at the links below. If the AWS account
+# you are using is already subscribed, this confirmation will be displayed on each page.
+# openSUSE Leap arm64 subscription: https://aws.amazon.com/marketplace/server/procurement?productId=a516e959-df54-4035-bb1a-63599b7a6df9
+# openSUSE leap amd64 subscription: https://aws.amazon.com/marketplace/server/procurement?productId=5535c495-72d4-4355-b169-54ffa874f849


Did we already subscribe to these in the vault_enterprise_ci account as well? I guess we'll find out with the ent PR

Not yet but I will do that with the ent PR.

I feel like we need to put this information in the scenario descriptions so it's easy to see. I'll make a note do that in my next PR that adds descriptions when I rebase on this.

enos/modules/install_packages/scripts/install-packages.sh

ryancragun

A couple small things I think we should clean up before merge but after lets push the green!

ryancragun · 2024-05-31T17:07:26Z

enos/enos-dev-scenario-pr-replication.hcl

@@ -139,7 +142,7 @@ scenario "dev_pr_replication" {
    // If you are using an ent edition, you will need a Vault license. Common convention
    // is to store it at ./support/vault.hclic, but you may change this path according
    // to your own preference.
-    vault_install_dir = matrix.artifact == "zip" ? var.vault_install_dir : global.vault_install_dir_packages[matrix.distro]
+    vault_install_dir = matrix.artifact == "zip" || matrix.artifact == "local" ? global.vault_install_dir["bundle"] : global.vault_install_dir["package"]


Based on this, it seems like we have no real use of vault_install_dir being a variable at all since we ignore it and use our global map. Seems like we should remove it entirely?

Our global map uses the artifact_type (bundle or package), which we don't have in this scenario, so this is like a re-mapping of our matrix.artifact to those.

ryancragun · 2024-05-31T17:12:13Z

enos/enos-globals.hcl

-    aws_region = ["us-east-1", "us-west-2"]
+    aws_region            = ["us-east-1", "us-west-2"]
+    distro_version_amzn2  = ["2"]
+    distro_version_leap   = ["15.4", "15.5"]


Great use of sample attrs. I'm still trying to figure out when it's better to use samples attrs or a matrix variant, but it feels like to me if it doesn't change logic it should be a sample attr going forward.

ryancragun · 2024-05-31T17:21:57Z

enos/modules/build_artifactory_artifact/locals.tf

-      "ent.fips1402"     = "vault-enterprise-fips1402_",
-      "ent.hsm"          = "vault-enterprise-hsm_",
-      "ent.hsm.fips1402" = "vault-enterprise-hsm-fips1402_",
+    amzn2 = {


ryancragun · 2024-05-31T17:22:19Z

enos/modules/disable_selinux/main.tf

+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: BUSL-1.1
+
+terraform {


This is my new favorite module in the entire codebase

ryancragun · 2024-05-31T17:22:30Z

enos/modules/disable_selinux/main.tf

+resource "enos_remote_exec" "make_selinux_permissive" {
+  for_each = var.hosts
+
+  # environment = {


Should we remove this comment?

ryancragun · 2024-05-31T17:30:16Z

enos/modules/install_packages/scripts/install-packages.sh

+  echo "Installing Dependencies: ${PACKAGES}"
+
+  # Use the default package manager of the current Linux distro to install packages
+  case $PACKAGE_MANAGER in


Massive improvement in our implementation here.

ryancragun · 2024-05-31T17:31:18Z

enos/modules/seal_pkcs11/main.tf

@@ -95,6 +95,7 @@ module "target" {
  ami_id          = module.ec2_info.ami_ids["arm64"]["ubuntu"]["22.04"]
  cluster_tag_key = local.id
  common_tags     = var.common_tags
+  disable_selinux = false


Huh, won't this make RHEL 9.3 not work with pkcs11 since we disable it for vault to start up there?

Ah, this was from earlier work, it's not needed now. Removed.

ryancragun · 2024-05-31T17:33:34Z

enos/modules/target_ec2_instances/variables.tf

@@ -24,6 +24,12 @@ variable "common_tags" {
  default     = { "Project" : "vault-ci" }
 }

+variable "disable_selinux" {


Since we modified this interface we should also update the other target modules so that if for some reason we decide to try spot instances again it should work.

ryancragun · 2024-05-31T17:34:21Z

enos/modules/vault_cluster/main.tf

@@ -15,6 +15,10 @@ terraform {
 data "enos_environment" "localhost" {}

 locals {
+  distro_version_sles = {


I don't see where we use this. Also, we should be getting this from variables, right?

yep this is leftover from earlier attempts. Removed.

ryancragun · 2024-05-31T17:35:17Z

enos/modules/vault_cluster/main.tf

@@ -37,6 +41,13 @@ locals {
    "pkcs11" = null
  }
  leader = toset(slice(local.instances, 0, 1))
+  netcat_command = {


I though we created this in globals. Why are we redefining it here?

In globals, we have a list of the packages that need to be installed for each distro:

distro_packages = { amzn2 = ["nc"] leap = ["netcat", "openssl"] rhel = ["nc"] sles = ["nc", "openssl"] ubuntu = ["netcat"]

We use a separate set of package names here for two reasons:

We are getting the name of the distro from enos_host_info, which returns slightly different distro names than the ones we use throughout the scenarios (e.g. opensuse-leap vs leap, amzn vs amzn2).

We include only the netcat CLI command used by the distro, as a single string. It seemed rather fragile to try to pick "the netcat one" from a list of multiple packages in our globals.

That makes sense. Too many maps!

ryancragun

I think testing each of the other target modules really quick and fixing up the disable section and then merge.

ryancragun · 2024-06-03T17:59:02Z

enos/modules/target_ec2_fleet/main.tf

+  count  = var.disable_selinux == true ? 1 : 0
+
+  hosts = { for idx in range(var.instance_count) : idx => {
+    public_ip  = aws_instance.targets[idx].public_ip


Did we test this change? I feel like this probably needs to be data.aws_instance.targets....

Recording here that we discovered some issues with the target_ec2_fleet module so will skip adding disable_selinux to this and the target_ec2_spot_fleet module for now, since we are not actively using either of those.

ryancragun · 2024-06-03T17:59:32Z

enos/modules/target_ec2_spot_fleet/main.tf

+  count  = var.disable_selinux == true ? 1 : 0
+
+  hosts = { for idx in range(var.instance_count) : idx => {
+    public_ip  = aws_instance.targets[idx].public_ip


Same goes for this instance

ryancragun · 2024-06-03T17:59:49Z

enos/modules/vault_cluster/main.tf

@@ -37,6 +41,13 @@ locals {
    "pkcs11" = null
  }
  leader = toset(slice(local.instances, 0, 1))
+  netcat_command = {


That makes sense. Too many maps!

…ios and modules Add Consul edition support to Enos scenarios and modules Add Linux distros and Consul edition to Enos samples Bump RHEL versions to 9.3 and 8.9

rebwill requested a review from a team as a code owner March 15, 2024 18:47

github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Mar 15, 2024

rebwill added pr/no-changelog pr/no-milestone labels Mar 15, 2024

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from df70052 to a6d67d1 Compare March 21, 2024 16:32

jasonodonnell reviewed Mar 22, 2024

View reviewed changes

enos/enos-scenario-proxy.hcl Outdated Show resolved Hide resolved

jasonodonnell reviewed Mar 22, 2024

View reviewed changes

enos/modules/ec2_info/main.tf Outdated Show resolved Hide resolved

jasonodonnell reviewed Mar 22, 2024

View reviewed changes

enos/modules/vault_artifactory_artifact/locals.tf Outdated Show resolved Hide resolved

jasonodonnell reviewed Mar 22, 2024

View reviewed changes

enos/modules/vault_verify_raft_auto_join_voter/scripts/verify-raft-auto-join-voter.sh Show resolved Hide resolved

jasonodonnell approved these changes Mar 22, 2024

View reviewed changes

ryancragun reviewed Mar 25, 2024

View reviewed changes

ryancragun reviewed Apr 11, 2024

View reviewed changes

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch 2 times, most recently from 54d078d to feba8cd Compare April 12, 2024 19:03

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from feba8cd to 6d51387 Compare April 30, 2024 19:02

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch 5 times, most recently from 75d5c76 to 7d4a65a Compare May 22, 2024 20:04

rebwill added backport/1.16.x backport/ent/1.15.x+ent Changes are backported to 1.15.x+ent labels May 22, 2024

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from 7d4a65a to 273d624 Compare May 23, 2024 18:28

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from 273d624 to a7c0e1a Compare May 30, 2024 20:03

ryancragun approved these changes May 31, 2024

View reviewed changes

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from 6a14482 to a7559f0 Compare June 3, 2024 15:52

ryancragun reviewed Jun 3, 2024

View reviewed changes

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from be777a0 to a8151e5 Compare June 4, 2024 19:15

Add Amazon Linux, openSUSE Leap, and SUSE SLES support to Enos scenar…

b6c82dd

…ios and modules Add Consul edition support to Enos scenarios and modules Add Linux distros and Consul edition to Enos samples Bump RHEL versions to 9.3 and 8.9

rebwill force-pushed the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch from 675d40d to b6c82dd Compare June 5, 2024 15:39

rebwill added the backport/1.17.x label Jun 5, 2024

rebwill merged commit c287395 into main Jun 5, 2024
91 checks passed

rebwill deleted the rebwill/add-additional-linux-distros-and-consul-editions-to-enos-scenarios branch June 5, 2024 16:58

This was referenced Jun 5, 2024

Backport of Add additional Linux distros and Consul editions to Enos scenarios into release/1.16.x #27361

Closed

Backport of Add additional Linux distros and Consul editions to Enos scenarios into release/1.17.x #27362

Merged

Add additional Linux distros and Consul editions to Enos scenarios #25983

Add additional Linux distros and Consul editions to Enos scenarios #25983

Conversation

rebwill commented Mar 15, 2024 • edited Loading

github-actions bot commented Mar 15, 2024 • edited Loading

github-actions bot commented Mar 15, 2024 • edited Loading

jasonodonnell left a comment

Choose a reason for hiding this comment

ryancragun left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rebwill commented Apr 5, 2024

ryancragun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryancragun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryancragun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rebwill commented Mar 15, 2024 •

edited

Loading

github-actions bot commented Mar 15, 2024 •

edited

Loading

github-actions bot commented Mar 15, 2024 •

edited

Loading

ryancragun left a comment •

edited

Loading