Proposal: Update yum repository layout to be per repository

[ehelms]

Background

Currently the yum.theforeman.org repository layout follows an almost common model with Foreman being the exception:

client/<version>
client/nightly
client/latest
katello/<version>/katello
katello/<version>/candlepin
katello/nightly/katello
katello/nightly/candlepin
latest/
nightly/
plugins/<version>
plugins/nightly
plugins/latest
pulpcore/<version>
releases/<version>

Where:

• top level latest/ points to Foreman repository
• top level nightly/ represents Foreman nightly repository
• top level 'releases/' represents Foreman releases
• All other repositories are under a directory that is first the name of the repository

Proposal

That the repositories get aligned under a common naming and structure to make them more predictable, easier to code against and allow better matching with stagingyum and our layout that is coming from Copr.

client/<version>
client/nightly
client/latest
foreman/<version>
foreman/nightly
foreman/latest
katello/<version>
katello/nightly
katello/latest
candlepin/<version>
candlepin/nightly
candlepin/latest
plugins/<version>
plugins/nightly
plugins/latest
pulpcore/<version>

In addition to this being the structure, we would retain (for some period of time) legacy symlinks:

releases/ -> foreman/
latest/ -> foreman/latest

[ekohl]

I'm wondering if we should instead go for versions as top level, then the version specific repos. So:

$ mkdir -p {3,8,3.9,nightly}/{client,foreman,plugins,katello}
$ ln -s 3.8 latest
$ tree
.
├── 3.8
│   ├── client
│   ├── foreman
│   ├── katello
│   └── plugins
├── 3.9
│   ├── client
│   ├── foreman
│   ├── katello
│   └── plugins
├── latest -> 3.8
└── nightly
    ├── client
    ├── foreman
    ├── katello
    └── plugins

We can also store the per-release GPG key in the directory (which would be missing in nightly). That way you immediately know which Katello release belongs to which Foreman release.

[ehelms]

What would you do with Pulpcore and Candlepin? For consistency, should we do by project and then version? e.g.

├── candlepin
│   ├── 4.1
│   └── 4.2
├── foreman
│   ├── 3.8
│   │   ├── client
│   │   ├── foreman
│   │   ├── katello
│   │   └── plugins
│   ├── 3.9
│   │   ├── client
│   │   ├── foreman
│   │   ├── katello
│   │   └── plugins
│   └── nightly
│       ├── client
│       ├── foreman
│       ├── katello
│       └── plugins
└── pulpcore
    ├── 3.22
    ├── 3.28
    └── nightly

[ekohl]

Yes, that would be best. Though for candlepin aren't they essentially also tied to a Foreman release? Today they're for a Katello release and Katello is essentially for a Foreman release. Or do you plan to maintain it more like pulpcore? Which would also imply a pulpcore-packaging equivalent.

Maybe that's a good rule: -packaging repository gets a top level directory. Each rpm/ branch in that repository is the second level directory, where rpm/develop maps to rpm/nightly. Each packages/* directory within that branch gets a third level directory.

Pulpcore doesn't have rpm/develop so there wouldn't be a pulpcore/nightly. It also doesn't split packages into multiple sub-directories so it doesn't get a third level directory.

[ehelms]

Yes, that would be best. Though for candlepin aren't they essentially also tied to a Foreman release? Today they're for a Katello release and Katello is essentially for a Foreman release. Or do you plan to maintain it more like pulpcore? Which would also imply a pulpcore-packaging equivalent.

Given it's one package I don't intend to management in the same way as our packaging repository. But I do think it would be convenient for us to treat Candlepin as stand-alone and released based on it's version to allow updates to Candlepin that can then be tested without the risk of breaking nightly (or a release branch). We could push updates to Candlepin, run for example our puppet test suite or something equivalent and know the package changes are "good" before we promote them into Katello.

Pulpcore doesn't have rpm/develop so there wouldn't be a pulpcore/nightly. It also doesn't split packages into multiple sub-directories so it doesn't get a third level directory.

Doesn't change things, just an FYI, as of today, this is no longer true -- https://github.com/theforeman/pulpcore-packaging/tree/rpm/develop
@Odilhao is working on nightly builds for Pulpcore

[ekohl]

I'm mostly trying to find a good way to describe a "policy".

Given it's one package I don't intend to management in the same way as our packaging repository. But I do think it would be convenient for us to treat Candlepin as stand-alone and released based on it's version to allow updates to Candlepin that can then be tested without the risk of breaking nightly (or a release branch). We could push updates to Candlepin, run for example our puppet test suite or something equivalent and know the package changes are "good" before we promote them into Katello.

Candlepin has always been an exception. It has never had its own specs in our packaging repository. In that sense, it's not surprising it needs its own policy. I'm not opposed to this. Taking it to puppet-candlepin, we could easily have a policy very similar to puppet-pulpcore. If the repo can be consumed standalone, that would simplify testing.

Doesn't change things, just an FYI, as of today, this is no longer true -- https://github.com/theforeman/pulpcore-packaging/tree/rpm/develop

Ah, I missed that. Good to know.

[ehelms]

Candlepin has always been an exception. It has never had its own specs in our packaging repository. In that sense, it's not surprising it needs its own policy. I'm not opposed to this. Taking it to puppet-candlepin, we could easily have a policy very similar to puppet-pulpcore. If the repo can be consumed standalone, that would simplify testing.

That is the tactic I am taking starting from the Copr side as there will be a dedicated build repo for it -- https://copr.fedorainfracloud.org/coprs/g/theforeman/foreman-candlepin-nightly-staging/

[ehelms]

I thought more about, and started testing the Candlepin part and realized it needed it's own repository even if that repo was realllly small and didn't actually have packages. To that end, I created https://github.com/theforeman/candlepin-packaging to allow me to work through this concept.

[evgeni]

• make a new rpm.tfm.o host and start "fresh" with an el9 box

[ehelms]

@evgeni @ekohl As we are changing the structure of the repositories going forward, and creating a new host, this leaves the question of what do we do about the old repositories, and hostname?

1. Create a separate dedicated host for the older releases that points at yum.theforeman.org and copy everything over
2. Create a vhost on the new rpm-repo machine to represent yum.theforeman.org and copy over the releases to serve as an archive
3. Keep el7 based web01 around as is until 3.12 is no longer supported and then delete all unsupported releases

[ekohl]

Keep el7 based web01 around as is until 3.12 is no longer supported and then delete all unsupported releases

This will break all upgrades for users, making it very painful for them.

Create a vhost on the new rpm-repo machine to represent yum.theforeman.org and copy over the releases to serve as an archive

I lean to this, though I'd probably try to use symlinks instead of copying.

[ehelms]

I lean to this, though I'd probably try to use symlinks instead of copying.

The copy here meant "from web01 to rpm-repo01" so that we can decommission web01.

[ekohl]

Then yes, I'd keep yum.theforeman.org as a vhost for some time. Worst case that'll be indefinitely. Probably best to see how much traffic it receives and make noise that people should migrate.