Rebuild Foreman infrastructure

[ekohl]

This is to address the Big Update where Foreman is updated from version 1.24 to 3.1. 3.1 is chosen because it's the last version that can update the DB from pre-2.0 databases and also runs on EL8.

It also moves to a Hiera-based structure to assign classes rather than the Foreman ENC. This makes it easier to bootstrap a new environment.

[ehelms]

It also moves to a Hiera-based structure to assign classes rather than the Foreman ENC. This makes it easier to bootstrap a new environment.

🎆

[ekohl]

What works:

• Foreman is installed and operational on an IPv6-only server
• Puppetserver runs on a dual stack server and reports back
• There's a (documented) path to bootstrap the whole infrastructure from scratch

I also copied the SSH keys from the existing Puppetserver because it allowed me to verify deployment. /usr/share/foreman/config/initializers/encryption_key.rb was also needed.

What remains to be done:

• Today SSH runs on a non-standard port to avoid a lot of noise. Do we want to do the same again?
• Current server also houses the shared secret storage. Move that to puppet01 now?
• Service name puppet.theforeman.org or puppetserver.theforeman.org? Keep puppet.
• Run the actual migration

Actual migration plan:

• Merge Only deploy Puppet on the new Foreman infra jenkins-jobs#283
• Merge this PR
• Stop services on puppetmaster.theforeman.org: systemctl stop httpd foreman.service foreman.socket dynflow\* foreman-proxy puppetserver
• Disable puppet on puppetmaster.theforeman.org: puppet agent --disable 'Migrating to new infrastructure'
• Copy Foreman DB
  • On old server: sudo -u postgres pg_dump -Fc foreman > foreman.pg_dump
  • On new Foreman server:

sudo /opt/puppetlabs/bin/puppet agent --disable 'Migrating to new infrastructure'
sudo systemctl stop foreman.service foreman.socket dynflow\*
sudo -u postgres dropdb foreman
sudo -u postgres createdb --owner foreman --encoding UTF-8 --locale en_US.UTF-8 foreman
sudo -u postgres pg_restore -d foreman foreman.pg_dump 
sudo foreman-rake db:migrate
sudo foreman-rake db:seed
sudo /opt/puppetlabs/bin/puppet config set --section agent environment production
sudo /opt/puppetlabs/bin/puppet agent --enable
sudo /opt/puppetlabs/bin/puppet agent -t

• Change the environment on the new Puppetserver:

sudo /opt/puppetlabs/bin/puppet config set --section agent environment production
sudo /opt/puppetlabs/bin/puppet agent -t

• Verify the host reports correctly.
• Migrate secrets
  • Copy /srv/secretsgit to new Puppetserver
  • Merge Use new secret storage theforeman-rel-eng#171
• Set up autosign records for all hosts
• Migrate machines by running a REX job (from the new Foreman):

if [[ -f /etc/redhat-release ]] ; then
    yum -y remove puppet6-release
    yum -y install https://yum.puppet.com/puppet7-release-el-7.noarch.rpm
    yum -y update puppet-agent
elif [[ -f /etc/debian_release ]] ; then
  apt -y remove puppet6-release
  wget https://apt.puppet.com/puppet7-release-bullseye.deb
  apt -y install ./puppet7-release-bullseye.deb
  apt update
  apt -y upgrade
fi
rm -rf /etc/puppetlabs/puppet/ssl
puppet config server puppet.theforeman.org
puppet ssl bootstrap
puppet agent -t

• Verify all nodes have reported in
• Remove autosign records
• Close Migrate Puppetserver to stand-alone instance #1685 & Migrate Foreman instance to new infrastructure #1686 once it's considered stable.

Now the old server can be considered old and should only remain as a fallback. It shouldn't be used by anything anymore. It's best to shut it down, but keep the old server for a while (2 - 4 weeks). Also keep the 1.24 database dump since we may want to use it for automated upgrade testing.

Foreman is running on 3.1. #1676 is a separate issue, but can be done after this is done.

[evgeni]

Can you elaborate a bit on this? You want to build a foreman01.$hoster.theforeman.org with a SAN of foreman.theforeman.org and need this setting to allow it?

Please be aware that someone (cough) found that this can open up a rather big can of worms (nothing checks the SAN when signing) and is thus not recommended by Puppet: "Be aware that enabling the setting could allow agent nodes to impersonate other nodes (including the nodes that already have signed certificates). Consequently, you must carefully inspect any CSRs with SANs attached."

Especially, Foreman doesn't show any SANs at all in the signing UI.

[evgeni]

(I guess we can enable that, sign the cert for foreman.tfm.o and disable it again?)

[ekohl]

You're right about that. The alternative I thought about was using Let's Encrypt for a proper URL. Tricky part there is access to it. Right now I have set up a firewall to only allow port 443 from the Puppetserver and my home network. I guess we could set up rules so that it does IP checks in Apache for everything except /.well-known or use DNS validation, but that's a lot trickier.

[evgeni]

I think I'd go with the easiest option, which to me seems to be "enable, sign, disable, (forget it 5 years)"?

[ekohl]

I'll opt to keep this in bootstrap for now, then when Puppet actually runs it should be disabled again.

[evgeni]

• Today SSH runs on a non-standard port to avoid a lot of noise. Do we want to do the same again?

If we ensure only key login is possible (which is true today, well, it also allows gssapi which we could disable, but no passwords), I don't think the "special" port adds any real security.

• Current server also houses the shared secret storage. Move that to puppet01 now?

I guess that's fine, yeah.

[evgeni]

I left a few comments, but overall the plan sounds good to me!

[ekohl]

Then the various machines need to be updated to the new Puppetserver. This will need work since there's a new CA. Additionally, they should be upgraded from Puppet 6 to 7.

At this point we should also remove all Puppet classes from hostgroups so we don't get duplicate definitions.

[evgeni]

Then the various machines need to be updated to the new Puppetserver. This will need work since there's a new CA. Additionally, they should be upgraded from Puppet 6 to 7.

At this point we should also remove all Puppet classes from hostgroups so we don't get duplicate definitions.

Is that required? I thought both HGs and the site.pp you have here end up being include and those can come multiple times?

[ekohl]

IIRC site.pp is evaluated first and then the class is defined. Since it tries to pass parameters it's a concrete class and that fails.

[ekohl]

I changed the post a bit to have easier copy-paste steps and checkboxes so it's easier to mark progress.

Remaining open discussion points:

• Certificate SAN handling: Rebuild Foreman infrastructure #1777 (comment)
• Sync service name to puppet profile? So rename puppet.t.o to puppetserver.t.o

I'll be out until Tuesday so hopefully we can perform the migration next week.

[evgeni]

FTR, I added a new host (mirror01.koji.aws.theforeman.org) to Puppet, but then realized I could have also used the new puppet setup?

[evgeni]

• Sync service name to puppet profile? So rename puppet.t.o to puppetserver.t.o

IMHO puppet is more descriptive, so I'd stick to that.

[evgeni]

For postgresql, I'd use pg_dump -Fc and pg_restore instead of a plain dump and psql to restore.

[ekohl]

Still need to migrate secrets and shut down the machine, but I mostly consider this done. Full notes from during the migration in https://community.theforeman.org/t/puppetserver-migration/31903/2. Short follow up action items:

• Use chrony on EL7 too #1803
• Rotate REX SSH key on puppet01 and client old puppetmaster entry #1804
• Modernize the module dgolja/golja-gnupg#38

[ekohl]

So it looks like backups are also sent to the old puppetmaster. For now I've created #1805.