Add stagingyum vhost to web01 for Copr staging repositories

[ehelms]

No description provided.

[ehelms]

This would create the vhost but there is the question of how the content gets to the Vhost. In the case of Koji, web01 rsyncs the repositories to it via a special script.

In the case of Copr, where we are generating the repositories on a Jenkins node, I do not think that strategy will work. I think we would need to scp the repositories to web01 in that case. I don't think we do anything like this today to model this after exactly. My open questions:

1. Is SCP the right way to transfer?
2. How do we lock web01 down to just allow this SCP action when initiated from the Jenkins node?

[evgeni]

This would create the vhost but there is the question of how the content gets to the Vhost. In the case of Koji, web01 rsyncs the repositories to it via a special script.

In the case of Copr, where we are generating the repositories on a Jenkins node, I do not think that strategy will work. I think we would need to scp the repositories to web01 in that case. I don't think we do anything like this today to model this after exactly. My open questions:

1. Is SCP the right way to transfer?
2. How do we lock web01 down to just allow this SCP action when initiated from the Jenkins node?

(if anyone wants to see the "special" script, it's this one: https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/web/files/deploy-yumrepo.sh )

I think rsync-over-SSH is the right answer, because we already have infrastructure for that in place (Debian uses that to push the built packages from the builders to web01). See https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/web/manifests/vhost/deb.pp and https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/web/manifests/vhost/stagingdeb.pp

[ehelms]

Let's see if I got the hang of this with this update. I've also opened up a draft for the Jenkins job in order to align the changes necessary to pull this workflow off (theforeman/jenkins-jobs#349).

If I understand this correctly, initiating an rsync from Jenkins will actually initiate the deploy-stagingyum.sh script, where it will do the rsync to a temporary location and then copy it to the deployed location on web01.

This change will require me to create a user via our Foreman? Or just an SSH key? I assume this gets created via some action in Foreman but then will need to be added to Jenkins.

[evgeni]

this is missing something like

diff --git puppet/modules/profiles/manifests/web.pp puppet/modules/profiles/manifests/web.pp
index 9383587a..8fa6fa8d 100644
--- puppet/modules/profiles/manifests/web.pp
+++ puppet/modules/profiles/manifests/web.pp
@@ -60,4 +60,9 @@ class profiles::web (
     rsync_max_connections => $rsync_max_connections,
   }
   contain web::vhost::yum
+
+  class { 'web::vhost::stagingyum':
+    stable                => $stable,
+  }
+  contain web::vhost::stagingyum
 }

to actually load that class

[evgeni]

overall, this is working

(tested by a vagrant up web with the above patch, then generating a key and using it in ~yumrepostage/.ssh/authorized_keys like this:

command="/home/yumrepostage/bin/secure_yumrepostage" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3AhxArA3lkxcFuSMsf2UMCyNyHHKC/aTiist7L3zQCeEb55jAMsDi1ZjnQou07TWuhqayPmeEF3gUGp5SeRjUTJu+6AkSA9zsh1bY5akOboYU4WGlRkspSiBCLbzevJNvAuDGRYH/fLyadIRGHKweXRCiIYglDhEuBrVf8E3u33NLCgloj848mhFTAEiKt+UBlfvQqcKsGwA7hE6pIOgSVKkApIsgR0Bv7Bn7QSgn5bbhVWd8zpAten3+LYOTaFTIvypc6qu3Px814K1XdWYvtm6+S4rGkfq8LPE6VWo6k/+8F1VH8bGIA3e1RMn99kNvFgpL3wxp0V6wsCEj4Qj1 root@web

and then doing an rsync manually

[ekohl]

Probably still good to limit this to Jenkins nodes instead of all machines. You can probably limit it to host ~ node*.jenkins.*.theforeman.org. You can test it out in Foreman since it's just going a search on fact_values.

Though I do worry about pagination:

foreman-infra/puppet/modules/secure_ssh/manifests/receiver_setup.pp

Line 70 in 4a6c278

$ip_data = foreman::foreman('fact_values', $foreman_search, '20', lookup('foreman_url'), $api_user, $api_pass)

[ehelms]

There is only 11 nodes, so I think we are OK and this search works.

[ekohl]

If all have both IPv4 and IPv6 then 11 * 2 = 22, so we'd miss 2 IPs. Today we're OK because not all nodes are dual stack.

[ehelms]

Agreed. And this current definition keeps it in line with the yum vhost as well:

https://github.com/theforeman/foreman-infra/blob/master/puppet/modules/web/manifests/vhost/yum.pp#L31-L35

[ekohl]

I think this is correct now.

[ekohl]

One thing to note: for other things I've also taken the approach of manually adding this key to Jenkins as a real secret and then use the Jenkins built in functionality. Perhaps worth considering here too.

[ehelms]

I seem to be getting permission denied from Jenkins to web01:

08:26:07  + export 'RSYNC_RSH=ssh -i /home/jenkins/workspace/staging_key/rsync_yumstage_key'
08:26:07  + RSYNC_RSH='ssh -i /home/jenkins/workspace/staging_key/rsync_yumstage_key'
08:26:07  + /usr/bin/rsync -avPx foreman/nightly [email protected]:rsync_cache/foreman/nightly
08:26:08  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).