k3s/k3d authenticate to secure registry 401 UNAUTHORIZED

[the-scott-hand]

./k3d version
k3d version v4.4.6
k3s version v1.21.1-k3s (default)

Hi there, I am trying to configure my k3d cluster to pull images from a secure registry. I am running the command in the following way:

k3d cluster create --servers 1 mycluster -p "8069:80@loadbalancer" --volume $K3D_PATH/registries.yaml:/etc/rancher/k3s/registries.yaml --volume $K3D_PATH/my.pem:/etc/ssl/certs/my.pem

my registries.yaml is as follows

mirrors:
    my.registry.org:
        endpoint:
            - https://my.registry.org
configs:
    my.registry.org:
        tls:
            ca_file: "/etc/ssl/certs/my.pem"

without the ca file I get x509 certificate signed by unknown authority, so I assume that my registries.yaml is being used in some way. However, there seems to be no combination of ca_file cert_file and key_file that get me past the failed to pull and unpack image <image> failed to resolve reference: <image> "failed to authorize: failed to fetch anonymous token: unexpected status: 401 UNAUTHORIZED.

When i copy my.pem to /etc/docker/certs.d/my.registry.org/ as client.key and client.cert, I am able to docker pull as expected without logging in. however specifying these in the configs section and mounting them does not seem to enable k3s to pull images from the secure part.

Where am i going wrong? Can anyone help me configure this correctly?

[brandond]

Your registry appears to require authentication, but you're not providing any? Did you want to provide a username and password, or cert_file and key_file for certificate auth?

[brandond]

I'm not sure what's necessary to get these mounted into k3d, you might ask this same question over in the k3d repo.

[the-scott-hand]

To clarify, I am using a secure registry and authenticate using registered server certs rather than a username/password situation. When using docker, copying and pasting my cert into files client.key and client.cert and placing them in /etc/docker/certs.d/my.registry.org/ is enough to get docker to pull from the secure registry. How can I configure my registries.yaml to accomplish this in k3s? I have experimented with using cert_file and key_file in the registries.yaml with this pem but I get the same error.

As far as mounting them into k3d, that is accomplished via the --volume $K3D_PATH/my.pem:/etc/ssl/certs/my.pem flag which I have confirmed I can do successfully, I didnt intend to ask any questions about mounting the certs.

Thanks for the quick response!

[brandond]

Well, the docs are at https://rancher.com/docs/k3s/latest/en/installation/private-registry/#configs - if cert_file and key_file don't work then I'm not sure what to suggest. This is all just passed through directly to containerd, which does the actual image pulling.

Can you get this to work properly with k3s? If so then it's probably a k3d issue and not a problem with how you're setting up registries.yaml

[the-scott-hand]

Hmmm ok I will try asking in the k3d git discussions, I wasnt sure which place to ask. Thanks!

[ChristianCiach]

According to your question over at k3d-io/k3d#705, it looks like you're trying to use the same file for all three attributes: ca_file, key_file and cert_file. I wouldn't be surprised if containerd just doesn't support this.

Try to open your my.pem with a text editor and split this file into three separate files.

[the-scott-hand]

and yes my toml does include all the references to the tls config. I was able to try adding the auth section with username and password and that worked, but I was hoping to not use a user/password combo in this file.

[the-scott-hand]

multiple users with sudo access on the same server with different docker accounts, trying to avoid putting their passwords in plaintext. wanted to run multiple instances of k3d. The server cert registered with the registry was working well for us with docker to solve this issue, and it seems ctr is able to pull the images when given those flags from inside the k3d container but just not k3s?

[the-scott-hand]

any ideas on how i can debug why ctr can pull my image from inside the container but kubernetes cant?

[iwilltry42]

@the-scott-hand , so if you add the files in the registries.yaml, they show up correctly in the containerd config toml?
Does ctr image pull without the tlscacert/tlscert/tlskey flags work then or not?

[the-scott-hand]

the ctr image pull without the flags does not work. Im not super familiar with the config.toml but it populated with values from my registries yaml

[the-scott-hand]

Not sure which of my many changes fixed the issue, but i got it working with the certs. These are some differences:

I switched to using the --config flag and specified all of my volumes and ports in that config file
My mounts previously had nodeFilter server[0] where I now have server[*] and agent[*] (configured for 1 server no agents)
The loadbalancer is currently disabled and wasnt previously
traefik is also disabled and wasnt previously
I have mounted my hosts /etc/resolv.conf onto /etc/resolv.conf in k3d
No longer mounting a "registries.yaml" just specified in the config file

Thanks for the responses

[brandond]

Yeah, you would probably want the registries configuration on all the servers and agents right? They all pull images...

[the-scott-hand]

yeah definitely, I just only had 1 server and no agents so i assumed server[0] was the full list