feat: dnf module

[xynydev]

A dnf module will eventually replace the rpm-ostree module. We should start investigating usage of dnf5 on atomic Fedora systems to determine a reasonable timeline for development of such a module.

https://fedoramagazine.org/whats-new-for-fedora-atomic-desktops-in-fedora-41/#first-step-towards-bootable-containers-dnf5-and-bootc

[fiftydinar]

It functions similarly to rpm-ostree when I tested it.

The only advantage I see is that we can enable copr repos with simpler command dnf copr enable maintainer/repo instead of manual curls.

Basically, when we would replace rpm-ostree occurences with dnf in the current module, users wouldn't notice any change in functionality except a bit different logs.

[xynydev]

Basically, when we would replace rpm-ostree occurences with dnf in the current module

I would rather create a new module called dnf. We may keep most of the configuration and even code the same, but there's no reason to keep the legacy name of rpm-ostree around.

[fiftydinar]

I'm in for making a new dnf module & announcing it to the users in blog, Discord, Mastodon & BlueSky (if it's used?).

Current structure from rpm-ostree module looks good & can be ported to dnf, except there needs to be some minor corrections in recipe format & some code logic replacements are needed.

Minor recipe format corrections

Repos

Currently, in rpm-ostree we just download the repos with curl directly to /etc/yum.repos.d/.

dnf has copr plugin to download & enable COPR repos.

• We can continue to use the same method of downloading repos using curl to /etc/yum.repos.d/
• Or switch to more native method of using dnf5 -y copr enable user/package
  • In case of dnf5 copr command usage, we can either
    • keep the same structure of repos array in recipe, so when it detects that it's copr URL, it will use dnf5 copr command
    • same as above, but users can just name the array copr: user/package instead of the full URL & we will detect that (this one is the best approach imo).
    • or add the new YAML array called repos-copr or similar, where users would just input user & package

Replacing packages

Currently, we use rpm-ostree override replace, which supports replacing packages from repo, direct URL & maybe local RPMs too.

It seems that there is no alternative for replacing packages in dnf.

dnf5 swap "${rem_packages}" "${inst_packages}" only does
rpm-ostree override remove "${rem_packages}" --install "${inst_packages}".

Everything else

Everything else looks relatively native.

[tulilirockz]

dnf5 has behavioural differences from rpm-ostree, as it probably does not install Recommends: packages by default! I ended up figuring that out when I was testing out the dnf5 migration on https://github.com/ublue-os/main and https://github.com/ublue-os/bluefin.

[tulilirockz]

rpm-ostree override replace can be replaced by dnf -y remove "${rem_packages}" && dnf -y install "${inst_packages}"

[tulilirockz]

Moving to dnf copr should be completely fine, too! There are some repos that sadly do not work rn like the ublue-os/akmods one that gets funky w/ the dnf copr behaviour (see: ublue-os/bluefin#1954)

[fiftydinar]

dnf5 has behavioural differences from rpm-ostree, as it probably does not install Recommends: packages by default! I ended up figuring that out when I was testing out the dnf5 migration on https://github.com/ublue-os/main and https://github.com/ublue-os/bluefin.

According to dnf defaults, the default is to install weak dependencies by default, just like rpm-ostree.
install_weak_deps=True

If  enabled, when a new package is about to be installed, all packages linked by weak dependency relation
(Recommends or Supplements flags) with this package will be pulled into the transaction.

I also personally didn't notice any difference in this behavior when I tested it a bit.

rpm-ostree override replace can be replaced by dnf -y remove "${rem_packages}" && dnf -y install "${inst_packages}"

That's basically what dnf swap does (except swap only supports swapping 1 package as I see now...).

But if there is no alternative, then sure, that can be also used & adjusted to match the replace behavior from recipe, without user noticing any difference.

Moving to dnf copr should be completely fine, too! There are some repos that sadly do not work rn like the ublue-os/akmods one that gets funky w/ the dnf copr behaviour (see: ublue-os/bluefin#1954)

Interesting, thanks for noting.

[fiftydinar]

I noticed in this Bazzite's PR:
https://github.com/ublue-os/bazzite/pull/1905/files

That they are using dnf5 -y upgrade --repo for replacing packages from the repo, so I think we found the command replacement for this actually.

[xynydev]

dnf has copr plugin to download & enable COPR repos.

I think we could do this simple change:

• keep the repos: array
• for each element, check if it starts with http(s)://, and use the curl method if so
• if not, give the repo to dnf5 -y copr enable and see what happens
  • if it was configured correctly, the repo should be enabled, but if there's a problem it should error out and we can print a nice error message in the logs

[fiftydinar]

• if not, give the repo to dnf5 -y copr enable and see what happens

I think a flaw in this method is support for repo files which are directly provided in /files/rpm-ostree/

So I think that my suggestion of using copr: user/package format is better.

[fiftydinar]

@gmpinder We will probably need to add --mount=type=cache,dst=/var/cache/libdnf5 in Containerfile when this gets merged

[gmpinder]

@gmpinder We will probably need to add --mount=type=cache,dst=/var/cache/libdnf5 in Containerfile when this gets merged

Sounds good, that'll be an easy addition.

[xynydev]

I think a flaw in this method is support for repo files which are directly provided in /files/rpm-ostree/

Oh right, I forgot that. If we assume copr repo names don't include the .repo suffix, but repo files all do, we use that as another way to detect it.

Adding another key would work too I guess, but then we'd have to debate the whole config structure related to that again lol.

[fiftydinar]

We should hold a bit releasing dnf module until all issues are resolved with it. rpm-ostree should be still well supported for F42 at minimum.

ublue-os/bluefin#1954 (comment)

Draft PR can be made though, to get us more ready.

[gmpinder]

So I think that my suggestion of using copr: user/package format is better.

I'm still of the opinion that we should have a separate top-level copr property that takes a list. Right now it's coded to be in the repos section and requires the user to define it like COPR repo/namespace. This, imho, could get confusing for users and be error prone if extra whitespace is added. I think we should encode as much semantics into the spec as we can to allow for easier readability and validation.

[tulilirockz]

I am also with @gmpinder in this one. It's both kind of confusing and having some typing-strict way to check if the repo is a COPR would be much better. Like, I think the best way would be to do something like this

# (...)
repos:
    - name: "myuser/chongus"
      copr: true # or something dunno
    - "other-regular-repo.repo"
    - anotherone.repo
# (...)

[fiftydinar]

I'm still of the opinion that we should have a separate top-level copr property that takes a list. Right now it's coded to be in the repos section and requires the user to define it like COPR repo/namespace. This, imho, could get confusing for users and be error prone if extra whitespace is added. I think we should encode as much semantics into the spec as we can to allow for easier readability and validation.

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.

I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.

For repos:, those would be enable, disable, changing priority & maybe something else too.

I am also with @gmpinder in this one. It's both kind of confusing and having some typing-strict way to check if the repo > is a COPR would be much better. Like, I think the best way would be to do something like this

# (...)
repos:
    - name: "myuser/chongus"
      copr: true # or something dunno
    - "other-regular-repo.repo"
    - anotherone.repo
# (...)

I think that distinct copr: is better, as I proposed having repo manipulation as outlined above.

[gmpinder]

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.

I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.

For repos:, those would be enable, disable, changing priority & maybe something else too.

I'd be down with this. Especially if there are flows where you'd want to enable/disable repos.

[fiftydinar]

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.
I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.
For repos:, those would be enable, disable, changing priority & maybe something else too.

I'd be down with this. Especially if there are flows where you'd want to enable/disable repos.

I added separate copr: array now.

I'll add some manipulation options to copr: & repos: later. Recipe format will need to change a bit for this for those arrays, so any suggestion for that would be great & which options should be added.

[gmpinder]

So for the copr: property, we could have enable, disable, and remove.

type: dnf
copr:
  enable: # could also be 'add' since `dnf copr enable` is used for both adding and enabling
    - some/repo
  disable: # use `dnf copr disable $REPO`
    - disabled/repo
  remove: # use `dnf copr remove $REPO`
    - removed/repo

That way it matches what is available under the command.

$ dnf5 copr --help
Usage:
  dnf5 [GLOBAL OPTIONS] copr [OPTIONS] <COMMAND> ...

Description:
  Manage Copr repositories (add-ons provided by users/community/third-party)

Commands:
  list                          list Copr repositories
  enable                        download the repository info from a Copr server and install it as a /etc/yum.repos.d/*.repo file
  disable                       disable specified Copr repository (if exists), keep /etc/yum.repos.d/*.repo file - just mark enabled=0
  remove                        remove specified Copr repository from the system (removes the /etc/yum.repos.d/*.repo file)
  debug                         print useful info about the system, useful for debugging

Options:
  --hub=HOSTNAME                Copr hub (the web-UI/API server) hostname

[gmpinder]

As for the repo: section, we could have an add, enable, disable. We could add a remove, but that would require figuring out which repo file is the correct one. You might be able to figure that out with dnf repo info --json <repo_name> | jq -cr '.[0].repo_file_path'.

type: dnf
repos:
  add: # uses `dnf config-manager addrepo`
    - https://pkg.earthly.dev/earthly.repo
    - https://cli.github.com/packages/rpm/gh-cli.repo
  disable: # uses `dnf config-manager setopt "$REPO.enabled=false"`
    - earthly-stable
  remove: # uses `rm -f "$(dnf repo info --json $REPO | jq -cr '.[0].repo_file_path')"`
    - google-cloud-cli
  enable: # uses `dnf config-manager setopt "$REPO.enabled=true"`
    - fedora-steam
install:
  packages:
    - earthly
    - gh
    - google-cloud-cli
    - steam

For the disable, remove, and enable properties, you will have to require the name of the repo as read by dnf.

[gmpinder]

Another question would be, do we want to have removing/disabling of repos be done at the beginning of the script like with adding, or should we do it at the end so that you could:

1. Add the repo
2. Install the packages you want
3. Remove/disable the repo to not interfere with future calls

I personally think it would be cleaner for the kind of flows I've seen to remove/disable after installation of packages so that you could keep all of that logic contained in a single module call.

[xynydev]

It seems repo management is one major pain point to discuss before the implementation of the module can be finished.

Here are the features I would definitely like to support:

• Adding repos from http(s) URLs
• Adding repos from COPR
• Cleaning up repos after install

Here are the features that I am still unsure about:

• Adding repos from files local to the repo (can easily be done with the files module, by just placing the file in whichever directory corresponds to /etc/yum.repos.d/ in the git repo)
• Removing repos, disabling repos, enabling disabled repos (more complicated to script in a way that works for everyone than to just use a script snippet to do the specific thing you want)

Here are two solutions I can come up with right now:

A: Magic

type: dnf
clean-up-repos: true # whether to remove added repos after module call is done
repos: # these repos are added & enabled
  - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
  - custom/myrepo.repo # ends with '.repo', treated as local file
  - username/myrepo # does not match previous, contains one '/', treated as COPR repo

B: Straightforward (with nesting)

type: dnf
repos:
  clean-up: true
  copr:
    - username/myrepo
  file:
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file

B.1: All operations

type: dnf
repos:
  copr:
    - username/myrepo
    - name: username/myrepo2
      operation: remove

B.2: All operations

type: dnf
repos:
  copr:
    add:
      - user/myrepo
    disable:
      - user/myrepo2

[xynydev]

Another idea

type: dnf
packages:
  - repo: fedora
    install:
      - micro
  - repo: terra
     install:
       - prismlauncher 
  - repo: myuser/myrepo # COPR (same magic as described above or something ig)
     install:
       - custompackage1
       - custompackage2
     replace:
        - kernel

All repos are cleaned out by default after packages are installed. This feels like an intuitive separation to me, but also an opinionated restriction.