why do i need to pull images over and over again?

[samuelkees]

I need to load the same images over and over again. Is there a way to say k3s to “persist” the images, so that I don’t have to pull them again? When I run watch k3s crictl images I can see that the images are deleted after some time.

I tried to set image-gc-high-threshold and image-gc-low-threshold in
/etc/rancher/k3s/config.yaml, but did not work.

kubelet-arg=image-gc-high-threshold: "99"
kubelet-arg=image-gc-low-threshold: "98"

What I’m doing wrong?

[brandond]

Images are pruned when you get low on disk space on the image filesystem. Raising the GC thresholds isn't a great idea, you'll probably just run out of space and have a completely different set of problems. The best way to resolve this is just to use a larger disk on your node.

Also, that's not the right syntax for a yaml file. You'd want something like:

kubelet-arg:
- image-gc-high-threshold=99
- image-gc-low-threshold=98

If pulling images is slow, you may want to look into setting up a local registry mirror, where you can have more control over which images are cached and for how long.

[ChristianCiach]

You need to use /etc/rancher/k3s/config.yaml. But I agree with brandond that this seems to be a bad idea.

[zmansuroglu]

The best way to resolve this is just to use a larger disk on your node.

I'm wondering what rate of disk on the node is used by k3s for images. I'm currently trying to install an image of about 650M on a worker node. The disk usage on this node looks like this:

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  3.8G  1.8G  1.9G  49% /
devtmpfs        4.0M  4.0K  4.0M   1% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           787M   75M  713M  10% /run
tmpfs           2.0G     0  2.0G   0% /tmp
tmpfs           2.0G  9.2M  2.0G   1% /var/volatile
/dev/mmcblk0p1  2.0G   34M  2.0G   2% /boot
tmpfs           394M     0  394M   0% /run/user/0
shm              64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7bf57eceb0cd7589311bd31947f5bf25355b776243ed65fb21a4622dfcf5b5f0/shm

Deploying the image fails and results in DiskPressure, so the svclb-traefik pod is evicted. How can I tell k3s to add from free space on the disk to the image file system?

[brandond]

You have a 3.8gb root partition, that's tiny. Did you mean to make it so small?

[zmansuroglu]

It's a heterogeneous cluster with a variety of embedded platforms. This is a Xilinx board running aarch64 and petalinux. I just started with the sd card from the factory setup of the platform. Maybe I should expand the disk size, but I don't think that was the problem. I saw that the image wasn't code compatible with the platform. It's a problem that can be caused by docker's buildx on the build host or containerd on the worker node.

buildx didn't report any problems, even the base image is not available for the aarch64 architecture. On the k3s cluster, the image crashed and restarted several times, throwing a diskpressure event. But it was actually a consequent error.

[brandond]

ok. I'm not sure what the image arch has to do with it getting cleaned up under disk pressure but it sounds like you have it figured out?

[zmansuroglu]

For testing purposes, I tainted all but one of the worker nodes. After the pod crashes, it keeps restarting. This results in multiple instances of the pod on the node in different states, which triggers the DiskPressure event.

The question for me is why doesn't containerd recognize that the container is not available for the architecture and pull the container and try to start it. But this is an off-topic question here, I think.

Thank you for your valuable assistance.