It looks like the Spegel is not working.

[bkkoli]

Environmental Info:
K3s Version: v1.30.1+k3s1 (80978b5b)

Node(s) CPU architecture, OS, and Version: Linux k3s-05 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

Cluster Configuration: 3 servers, 0 agents

Describe the bug:

It looks like the Spegel is not working.

Steps To Reproduce:

To test the Embedded Registry function, the "ctr -n=k8s.io image import nginx.tar" command was used to import the latest nginx image to the first node.
And the "/etc/resolv.conf" of the second node was annotated to prevent Internet connection.
Later, the Pod was forced to appear on the second node, but an error occurred that the image could not be pulled.

• Installed K3s:

Expected behavior:

Pulling Image of nginx becomes normal.

Actual behavior:

An Error of fail about pulling image occurs.

Additional context / logs:

[brandond]

You haven't described taking any steps to enable or configure the embedded registry. Did you do so? Have you checked the service logs on your nodes to confirm that the embedded registry is starting and mirroring the registry that your pod's images are pulled from?

[bkkoli]

@brandond

You haven't described taking any steps to enable or configure the embedded registry. Did you do so? Have you checked the service logs on your nodes to confirm that the embedded registry is starting and mirroring the registry that your pod's images are pulled from?

First, I would like to say that I am sorry for the lack of detailed explanation.

1. I downloaded k3s from k3s-ansible repository and installed the v1.30.1+k3s1 version through ansible.
2. k3s "server args" are as follows. --flannel-backend=none --disable-network-policy --embedded-registry
3. /etc/rancher/k3s/registries.yaml files of all k3s nodes are as follows.

mirrors:
        "*":

4. The CNI of k3s is Cilium and was installed as the Default Option through Cilium cli.
5. To test the Embedded Registry function, the ctr -n=k8s.io image import nginx.tar command was used to import the latest nginx image to the first node.
6. the "/etc/resolv.conf" of the second node was annotated to prevent Internet connection.
7. the Pod was forced to appear on the second node, but an error occurred that the image could not be pulled.

The presence of Spegel in all nodes was confirmed through the following command.

• k3s-04 is first node.
• k3s-05 is second node.
• k3s-06 is third node.

root@k3s-04:/etc/rancher/k3s# kubectl get --raw /api/v1/nodes/k3s-04/proxy/metrics | grep -F 'spegel'
libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 6
spegel_advertised_image_digests{registry="quay.io"} 1
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 7
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 13
spegel_advertised_images{registry="quay.io"} 1
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 57
spegel_advertised_keys{registry="quay.io"} 11
# HELP spegel_mirror_requests_total Total number of mirror requests.
# TYPE spegel_mirror_requests_total counter
spegel_mirror_requests_total{cache="miss",registry="docker.io",source="internal"} 52
spegel_mirror_requests_total{cache="miss",registry="quay.io",source="internal"} 12
root@k3s-04:/etc/rancher/k3s# kubectl get --raw /api/v1/nodes/k3s-05/proxy/metrics | grep -F 'spegel'
libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 3
spegel_advertised_image_digests{registry="quay.io"} 1
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 3
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 6
spegel_advertised_images{registry="quay.io"} 1
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 20
spegel_advertised_keys{registry="quay.io"} 11
# HELP spegel_mirror_requests_total Total number of mirror requests.
# TYPE spegel_mirror_requests_total counter
spegel_mirror_requests_total{cache="hit",registry="docker.io",source="internal"} 7
spegel_mirror_requests_total{cache="miss",registry="docker.io",source="internal"} 13
spegel_mirror_requests_total{cache="miss",registry="quay.io",source="internal"} 12
root@k3s-04:/etc/rancher/k3s# kubectl get --raw /api/v1/nodes/k3s-06/proxy/metrics | grep -F 'spegel'
libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 2
spegel_advertised_image_digests{registry="quay.io"} 2
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 2
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 4
spegel_advertised_images{registry="quay.io"} 2
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 12
spegel_advertised_keys{registry="quay.io"} 18
# HELP spegel_mirror_requests_total Total number of mirror requests.
# TYPE spegel_mirror_requests_total counter
spegel_mirror_requests_total{cache="hit",registry="docker.io",source="internal"} 7
spegel_mirror_requests_total{cache="miss",registry="docker.io",source="internal"} 6
spegel_mirror_requests_total{cache="miss",registry="quay.io",source="internal"} 20

An error of the nginx pod is as follows.

[brandond]

nginx:latest

I'd missed that you were literally using the latest tag when you said the latest nginx image.

You should avoid using the latest tag, in favor of something less mutable. This is covered in the docs:
https://docs.k3s.io/installation/registry-mirror#latest-tag

[bkkoli]

@brandond
As you said, I changed the tag to a value other than 'latest', and it worked well. Thank you for your sincere advice.