Quadlet support for enable and templates

[ygalblum]

Purpose

This discussion comes to combine the discussions being held in two separate issues about Quadlet enable #17567 and supporting templates #17662. Since both issues present similar discussion points, I would like to explore them together.

The current state

Since systemd considers the service files created by Quadlet and as transient, they cannot be enabled by calling systemctl enable.
To overcome this obstacle, Quadlet automatically enables services when their corresponding Quadlet unit files contain any of the keys Alias, WantedBy or RequiredBy in their Install section.
In addition, Quadlet currently does not support systemd templates and as a result disregards the DefaultInstance key.

Automatic Enable

The problem

Automatically enabling services couples between the creator of the Quadlet unit file and its user in terms of when the service is enabled.
For example, when a package consists of an executable and a service unit file, the user may choose if and when the to enable the service. However, in the current Quadlet implementation, a package distributed using a container image and a Quadlet unit file will either get enabled as part of the installation or never (if none of the required keys are set).

Suggested solution

Here: #17567 (comment) @alexlarsson suggested adding a new command podman quadlet enable <Quadlet unit>. The question is where would the symlink created by the command should reside.
Currently, Quadlet creates the symlinks under its output directory. However, this output directory, in normal operations, is the runtime directory (system or user) which does not persist. So, Quadlet will have to recreate the symlinks. This raises the quesion, how will it know which units were enabled.

Possible solution: should the symlinks be created somewhere else? E.g. the config dirs as suggested here #17662 (comment)

Supporting Templates

Supporting systemd templates adds another complexity into the mix, but they have direct connection with enabling services and therefore should be addressed together.

DefaultInstace

Currently, Quadlet does not support templates at all and disregards the templating option of unit files with @ at the end of their name.
With the current implementation the solution would be to create the symlink with the value of the DefaultInstance after the @ sign. But, what's the point of a template that is used only once.

Multiple Instances

Here: #17662 (comment), @mfschumann suggested to add an new Instances key that will instruct Quadlet to create a symlink for each instance in Instances.
The problem is that such a solution keeps the coupling between the creator and the user of the service.

Questions for Discussion

Automatic Enabling

Should Quadlet keep the current behavior of automatic enabling of the services?
If not, how should the transient storage issue be addressed? Can we use a different (persistent) location? Store the list of enabled services somewhere?

Templates

Should Quadlet support templates?
If yes, The handling of DefaultInstace is strait forward. But, the handling of additional instances depends on the decision on the previous question.
If the behavior is kept, should Quadlet support an Instances key (maybe in a new Quadlet section)?
If not, then the solution would probably be a combination of the two solutions

[ygalblum]

@rhatdan @alexlarsson @vrothberg @mfschumann @MartinX3 WDYT?

[MartinX3]

I think it's a good idea to coordinate the work on the systemd support.

I just hope the two features can be delivered in different updates without delaying each other.

[ygalblum]

I think the order will have to be first to decide about enable as it dictates the set of options for the templates support. So, they will have to come one by one.

[mfschumann]

@ygalblum Thanks for unifying the two issues into one and summing up the discussion / questions. I agree that the two issues should be solved together.

[vrothberg]

Here: #17567 (comment) @alexlarsson suggested adding a new command podman quadlet enable . The question is where would the symlink created by the command should reside.

If we took this path, I think this should be Quadlet command, not a Podman command. Let's not leak Quadlet into Podman to have a clear separation of concerns.

Should Quadlet support templates?

What is the use case for using templates? It seems counterintuitive to the simplicity of Quadlet. I am not saying no but I a clear use case may be a better guidance in decision making than an open question.

Should Quadlet keep the current behavior of automatic enabling of the services?

I don't feel strongly either way as long as it can be changed.

If not, how should the transient storage issue be addressed?

Instead of creating transient units, can't Quadlet write the files to preset paths? For a rootless user, it would be $HOME/.config/systemd/user.

[mfschumann]

If not, how should the transient storage issue be addressed?

Instead of creating transient units, can't Quadlet write the files to preset paths? For a rootless user, it would be $HOME/.config/systemd/user.

As far as I understand, a main selling point for using quadlet instead of podman generate systemd is that the admin/user only has to specify the container config once and gets a systemd unit that automatically updates to the newest best practices for running podman as systemd service on each systemctl daemon-reload. This functionality is only straight-forward to realize if the generated units are transient: If the generated units would be in a persistent location, the admin/user would be free to modify them. That would make it difficult for quadlet to keep the units up to date without interfering with the changes made by the admin/user.

How about using the .container file to store persistent information for quadlet?

For example a new [Quadlet] section in the .container file could persist the information which units to enable:

[Quadlet]
EnabledUnits=foo.service

That would symlink /run/systemd/generator/default.target.wants/foo.service -> /run/systemd/generator/foo.service when running quadlet.

The syntax could easily be expanded to support instances of template units:

[Quadlet]
[email protected] [email protected]

This would symlink /run/systemd/generator/default.target.wants/[email protected] -> /run/systemd/generator/[email protected] and /run/systemd/generator/default.target.wants/[email protected] -> /run/systemd/generator/[email protected] when running quadlet.

The [Quadlet] section could be maintained by the admin/user, a quadlet enable command, or both.

[ygalblum]

How about using the .container file to store persistent information for quadlet?

While it will work, I'm not sure I like the idea of Quadlet editing the .container (or any of its other unit types) file. I mean, I don't love the idea of Quadlet editing its input file.

[mfschumann]

Think about it this way: There could be two different components that complement one another:

1. Quadlet, the systemd generator. That's what we have already and what does a good job converting .container files into systemd units.
2. Quadlet-specific utilities that assist the admin/user in dealing with .container files. In this component there could be a functionality that enables/disables generated units (like systemctl enable) or another function that generates .container files from a running container (like podman generate systemd).

I don't have strong feelings about wether the two components come as subcommands in a single binary or are completely separate tools. But I think that having this kind of utilities available would help people a lot getting started and working with quadlet.

[alexlarsson]

I don't think it is right to have the instantiation be part of the template. The point of templates is that they are easy to instantiate and tweak without having to modify the base image.

And, the way you instantiate and tweak a template file like [email protected] is that you create instances like [email protected] and then you set your custom options in these.

So, maybe we should just implement the same for quadlet. I.e. add support for [email protected] and generates a template instantiation service. This could just forbid changes to the [Container] section so we don't have to re-read the template file and combine them but just let systemd combine the non-podman stuff. If we had this, you could then disallow the [Install] section in the template file, but allow it in the instance files. Enabling a template would then just be adding an instance file.

[alexlarsson]

I mean, we could allow modification of [Container] in the template instantiation file if we wanted, but that would make the implementation much more complex, as it would need to read the base file and combine them. And the generated file would perhaps seem a bit weird once generated, as they would duplicate everything in the base file which is not how normal template service files look.

[ygalblum]

If we took this path, I think this should be Quadlet command, not a Podman command. Let's not leak Quadlet into Podman to have a clear separation of concerns.

I understand your point. However, Quadlet is a systemd-generator and as such adheres to the API defined for generators. So, I don't think we should use the same executable (even if we can make it work). Maybe something else? Maybe we should rename the current Quadlet executable (e.g. quadlet-generator) and create a new one for such operations.

clear use case may be a better guidance in decision making than an open question

@mfschumann gave an explanation to what he's trying to do here: #17662 (comment).
Examples I can think of for templates:

• In .kube the path to the configmap file
• The exposed port number
• An environment variable value

can't Quadlet write the files to preset paths?

As a systemd-generator, Quadlet gets as a parameter from systemd the directory to which it can write the generated unit files. It is not supposed to write them anywhere else.

[ygalblum]

Just to make sure I understand your suggesting. If I combine the two, quadlet while change to be a CLI tool with a generate command and the podman-system-generator will become a script that does quadlet generate $1 $2 $3. Right?

BTW such a change will also require moving quadlet's installation directory as today it's not in the PATH

[mfschumann]

Yep, that's what I had in mind.

[Luap99]

A symlink can still work when the binary just checks if argv[0] is quadlet or podman-system-generator

[ygalblum]

@Luap99 I'm not familiar enough with Cobra, can you route the command based on arg[0]?
But, even if you can, not sure I'd want to have the code depend on the name of the executable

[Luap99]

Not within cobra but is trivial to setup the check before you construct the cobra command structure.
We already make heavy use of this, check c/storage/pkg/reexec for example.

[alexlarsson]

The upstream systemd issue for better enable is:
systemd/systemd#28006

[alexlarsson]

Maybe #20828 can be extended to support this? I.e. have a template .container file and then use drop-ins for enabling individual instances?

[mfschumann]

AFAIK drop-ins usually allow for setting parameters outside of the main unit file. Introducing parameters that can only be specified in a drop-in (but not in the main unit file) would be somewhat against this philosophy.
Instead, I'd suggest to add something like a EnabledUnits parameter (#17744 (reply in thread)) that can then either be specified in the main .container file or in a drop-in. This would have the added benefit that it would not require any extensions to #20828.

[alexlarsson]

I didn't mean adding a new setting. The dropin file would just use the standard [Install] section which also works in the main file. The changes needed to #20828 would be more about which dropins are read. I.e. for a templated file we would also read instanced dropins.

But, I think the right thing is not to special case template instance dropins, but rather to actually support templat instance unit files. See my other comment.

[mfschumann]

I like this suggestion, that would be a practical way to go. Thanks for the clarification!

[alexlarsson]

I created a PR for this: #21068

[alexlarsson]

This is merged now

[mfschumann]

Cool! Can you tell in which release this will be shipped?

[rhatdan]

5.0

[rhatdan]

Later this month or early March.

[richiedaze]

I have been using quadlet .container templates to create named toolboxes and .image to change the container image of the toolbox. I can even enable them on login without enabling them the usual way. The DefaultInstance key does not work for me in regular systemd templates neither.

• podman version 4.9.0

• ~/.config/containers/systemd

[email protected]

# Quadlet container template for toolbox  
# copyright 2022 @richiedaze:fedora.im
# SPDX-License-Identifier: GPL-2.1-or-late

# [email protected]
# %i = instance
# %p = program
# %S = state

[Unit]
AssertFileIsExecutable=/usr/bin/toolbox
Description=Quadlet container template for toolbox
Documentation=man:podman-systemd.unit(5)

[Container]
# Quadlet Container settings

# Add Capabilities
AddCapability=CAP_SYS_ADMIN

# Autoupdate mode
AutoUpdate=registry

# The name of the container
ContainerName=%i-%p

# Environmental variables inside container.
Environment=TOOLBOX_NAME=%i-%p
Environment=TOOLBOX_PATH=/usr/bin/%p

# Initial command inside container.
Exec=%p init-container \
    --log-level info \
        --home %h \
        --shell %s \
        --user %u \
        --uid %U \
        --gid %G \
        --home-link \
        --media-link \
        --mnt-link
        
# Show logs of this Container in Debug mode.
#GlobalArgs=--log-level debug

# Name of the host in the Container.
HostName=%i-%p

# Location of the base image.
Image=toolbox.image

# Container's mounts
Mount=type=devpts,destination=/dev/pts

# The network of the container.
Network=host

# Features otherwise unsupported by the Quadlet generator.
PodmanArgs=--dns none
#PodmanArgs=--group-add keep-groups
PodmanArgs=--ipc host
PodmanArgs=--no-hosts
PodmanArgs=--pid host
#PodmanArgs=--privileged

# Disable security labels 
SecurityLabelDisable=true

# Rootless mappings
#
# Default mapping
#UIDMap=0:${UID}:1
#GIDMap=0:${GID}:1
#
#UIDMap=0:1${UID}
#GIDMap=0:1${GID}
#UIDMap=${UID}:0:1
#GIDMap=${GID}:0:1
#UIDMap=${UID}:${UID}:65536
#GIDMap=${GID}:${GID}:65536

# User preferences
User=root:root
UserNS=keep-id

# Volumes

# Bind Non-volatile Overlayered volumes onto the container.
#Volume=%i-%p-etc-dnf:/etc/dnf:O,upperdir=%S/%p/%i-%p/etc/dnf,workdir=%S/%p/%i-%p/workdir/1
Volume=%i-%p-usr:/usr:O,upperdir=%S/%p/%i-%p/usr,workdir=%S/%p/%i-%p/workdir/2
Volume=%i-%p-var:/var:O,upperdir=%S/%p/%i-%p/var,workdir=%S/%p/%i-%p/workdir/3

# Bind host's directories onto the container.
Volume=/dev:/dev:rslave
Volume=/run/user/%U:/run/user/%U
Volume=/run/media:/run/media:rslave

# Bind host's root directory onto the container.
Volume=/:/run/host:rslave

# Bind host's sockets onto the container.
Volume=/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket
Volume=/run/avahi-daemon/socket:/run/avahi-daemon/socket
Volume=/run/dbus/system_bus_socket:/run/dbus/system_bus_socket
Volume=/run/pcscd/pcscd.comm:/run/pcscd/pcscd.comm

# Bind host's toolbox profile and executable onto the container.
Volume=/etc/profile.d/%p.sh:/etc/profile.d/%p.sh:ro
Volume=/usr/bin/%p:/usr/bin/%p:ro

# Bind Home directory onto the container.

# 1. Bind default home directory.
#    Use home directory normally. (toolbox default)
#Volume=%h:%h:rslave

# 2. Bind overlay on custom skeleton directory.

# 2a. Temporary (volatile overlayer).
#     Use a new overlayered skeleton directory every time.
#Volume=%E/skel:%h:O

# 2b. Permanent (non-volatile overlayer).
#     Use saved overlayered skeleton directory.
Volume=%E/skel:%h:O,upperdir=%S/%p/%i-%p%h,workdir=%S/%p/%i-%p/workdir/4

# 3. Bind extra home directories
Volume=%h/.local/src:%h/.local/src:rslave

# Working directory inside the container.
WorkingDir=%h

[Service]
# Show logs of this Service in Debug mode.
#Environment=SYSTEMD_LOG_LEVEL=debug

# Clone missing home skeleton directory.
ExecStartPre=bash -c 'test -d %E/skel || \
    cp --recursive /etc/skel %E'

# Create Non-volatile overlayer directories.
ExecStartPre=-bash -c 'mkdir --parents \
    --context=system_u:object_r:container_file_t:s0:c1022,c1023 \
        %S/%p/%i-%p/etc/dnf \
        %S/%p/%i-%p/%h \
        %S/%p/%i-%p/usr \
        %S/%p/%i-%p/var \
        %S/%p/%i-%p/workdir/{1..4} '

# ??? Rebuild volumes after podman auto-update
#ExecStopPost=-podman volume rm --force \
#    %i-%p-etc-dnf \
#    %i-%p-usr \
#    %i-%p-var
    
# Extend Timeout to allow time to pull the image
TimeoutStartSec=900

# Working directory inside the service.
#WorkingDirectory=%S/%p/%i-%p/workdir/

[Install]
# Start on user login
#WantedBy=graphical-session.target

toolbox.image

# Quadlet image template for toolbox  
# copyright 2022 @richiedaze:fedora.im
# SPDX-License-Identifier: GPL-2.1-or-late

[Image]
# The image to pull.  It is recommended to use a fully qualified image name rather than a short name, both for performance and robustness reasons.
# The format of the name is the same as when passed to podman pull. So, it supports using :tag or digests to guarantee the specific image version.

Image=registry.fedoraproject.org/fedora-toolbox:39

[richiedaze]

administrator@fedora:~$ podman --version 
podman version 4.9.0
administrator@fedora:~$ systemctl --user start toolbox@projectatomic
administrator@fedora:~$ systemctl --user status toolbox@projectatomic
● [email protected] - Quadlet container template for toolbox
     Loaded: loaded (/var/home/administrator/.config/containers/systemd/[email protected]; generated)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
             /run/user/1000/systemd/user.control/[email protected]
             └─50-CPUWeight.conf, 50-IOWeight.conf
     Active: active (running) since Fri 2024-02-23 17:42:55 EST; 27s ago
       Docs: man:podman-systemd.unit(5)
    Process: 136656 ExecStartPre=bash -c test -d /var/home/administrator/.config/skel ||      cp --recursive /etc/skel /var/home/administrator/.config (code=exited, status=>
    Process: 136658 ExecStartPre=bash -c mkdir --parents      --context=system_u:object_r:container_file_t:s0:c1022,c1023          /var/home/administrator/.local/state/tool>
   Main PID: 136730 (conmon)
      Tasks: 7 (limit: 8881)
     Memory: 908.5M
        CPU: 21.613s
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/app-toolbox.slice/[email protected]
             ├─libpod-payload-2f00cc6079207fc52264be2fdebd3ee8bfa627b82fb1c5fc8d529871e20b9ca8
             │ └─136732 toolbox init-container --log-level info --home /var/home/administrator --shell /bin/bash --user administrator --uid 1000 --gid 1000 --home-link --me>
             └─runtime
               └─136730 /usr/bin/conmon --api-version 1 -c 2f00cc6079207fc52264be2fdebd3ee8bfa627b82fb1c5fc8d529871e20b9ca8 -u 2f00cc6079207fc52264be2fdebd3ee8bfa627b82fb1c>

[ygalblum]

The support for templates was not part of v4.9 and will be released in v5.0

[richiedaze]

@ygalblum ,What I am saying is that it already works. Try it

[ygalblum]

@richiedaze Sorry, I misread your comment.

The main issue of this discussion was the enable feature. Since Quadlet is a systemd generator the units it creates are considered transient and therefore the user can enable them by calling systemctl enable <service name>. To accommodate this restriction, Quadlet automatically enables units that include the [Install] section. But, it does so only once and as a result does not support templates.
If you are OK with starting them manually (even if it's part of the loading scripts), then yes it should work (unless somewhere Quadlet errs and overrides the %i sign)

[dontlaugh]

Howdy everybody. I am playing with .build units on my fedora machine. Version installed from Copr:

Client:       Podman Engine
Version:      5.2.0-dev-57fe4c43b
API Version:  5.2.0-dev-57fe4c43b
Go Version:   go1.22.4
Built:        Sun Jun 23 00:00:00 2024
OS/Arch:      linux/amd64

My question is: are %i template directives available in a [Container] stanza's Image= param?

Here is the contents of /etc/containers/systemd/

[email protected]
[email protected]
cro-services-website@dockerfile_update.build -> [email protected]
cro-services-website@dockerfile_update.container -> [email protected]

I'd like to specify a templated build as the Image= in a templated container. The build works fine, but the container template doesn't work. Here is the error message

quadlet-generator[16915]: converting "[email protected]": requested Quadlet image cro-services-website@%i.build was not found
converting "cro-services-website@dockerfile_update.container": requested Quadlet image cro-services-website@%i.build was not found

Here is the container template file

[Container]
Image=cro-services-website@%i.build

I can work around this, but should templating the Image= work? I adapted an example from the latest docs. https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

[ygalblum]

The problem is not with the usage of the template alone but with its usage along with a .build file.

The reason this fails is that Quadlet needs to translate the value to of the Image stanza for two purposes:

1. The name of the image produced by the referenced .build file to add to the podman command
2. The name of the service Quadlet will generate for the .build file to add to the service's dependency list.

BTW you will probably see similar issues if you use templates of other Quadlet files, e.g. .image for the same issues or .volume for slightly different ones.

Now, I'm not saying its impossible to implement it. Would you like to try?

[dontlaugh]

@ygalblum I don't think I can take this on in the near future. :( But you do seem to have targeted the issue. At the risk of being pedantic, I'm going to add some more thoughts here, to confirm my own understanding and in case it helps somebody else who comes along.

Here is my workaround: supplying the part of the image tag I am aware of, with the -build suffix that is generated by the .build file.

[Container]
# build suffix is required
Image=localhost/cro-website:%i-build

This involves a little more coupling, since I need to know the internals of the .build file instead of what is visible from the outside: the name of the templated symlink.

1. The name of the image produced by the referenced .build file to add to the podman command

The %i template directive automatically appends a "-build" suffix. For good reason, I think. Given these templates and symlink instantiations in /etc/containers/systemd/...

[email protected]
[email protected]
cro-services-website@dockerfile_update.build -> [email protected]
cro-services-website@dockerfile_update.container -> [email protected]

Running systemctl daemon-reload yields the following generated units, both templates and instantiation of those templates (where %i == dockerfile_update).

# systemctl list-unit-files
[email protected] generated       -       
[email protected]       generated       -       
cro-services-website@dockerfile_update-build.service generated       -       
cro-services-website@dockerfile_update.service       generated       -

Instantiations of the .build template append "-build" to %i, both internally in the template and in the resulting generated unit file name. This has the very nice property of having the same prefix names for .build and .container files (and presumably .volume and .image), which is very systemd-ish (e.g. .socket targets a .service)

So yea, I can see how you could make my original code work, and it would be great if it did.

My only hesitation would be some edge case or future use case that could be thwarted by enshrining this little bit of magic. I just haven't used templated units that much!

As a practical matter, this implicit transformation/rendering of %i in different context seems like something a unit test could specify first and let it soak. Maybe I'm just being paranoid, though.

[aradban]

Just two cents from a system administrator's standpoint. What I expect when I create a .container file is for it to behave like other unit files. Having system, socket, mount, etc. units behave one way, and container units another, is confusing.

It seems like, for the most part, the sysadmin shouldn't even need to know about quadlet. I'm just creating a container unit. I'm reading the blogs/docs on how this works, and I'm thinking, "That's great. They have container units now. But what's all this quadlet business." All I really need to know is that systemd creates services from a container file using a generator, and these services run as pods that I can interact with using podman. That fits cleanly into my existing systemd mental framework.

Otherwise, you get mired in a C++ sort of mess, with broken paradigms that require frequent searches to figure out why things don't work in the way that the paradigm intuitively leads me to expect them to work.