Add support for pinning the infra container image

[afbjorklund]

KEP 2040: Kubelet CRI support

https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2040-kubelet-cri#pinned-images

• Clean up dockershim flags in the kubelet kubernetes/kubernetes#106893

Requires CRI API 1.23

[hakman]

@evol262 Maybe this should not be closed yet?

[afbjorklund]

As far as I know, this is what the --pod-infra-container-image option to cri-dockerd is (still) doing.

[evol262]

Ah, sort-of. --pod-infra-container-image is more of "use this image for pause". Docker itself doesn't proactively garbage-collect/"prune" images, so this was left open until when/if there's a story around it.

[afbjorklund]

Seems rather straight-forward, once the API is updated from 0.22.8 to 0.23.0 in order to add "Pinned"

 // ImageSpec is an internal representation of an image.
 message ImageSpec {
     // Container's Image field (e.g. imageID or imageDigest).
@@ -1130,6 +1247,10 @@ message Image {
     string username = 6;
     // ImageSpec for image which includes annotations
     ImageSpec spec = 7;
+    // Recommendation on whether this image should be exempt from garbage collection.
+    // It must only be treated as a recommendation -- the client can still request that the image be deleted,
+    // and the runtime must oblige.
+    bool pinned = 8;
 }
 
 message ListImagesResponse {

Then again, this means also having to implement PodSandboxStats and ListPodSandboxStats functions...

[afbjorklund]

They are also bumping the default version of registry.k8s.io/pause to match k8s, from 3.6 to 3.9

For the pinned, it seemed enough to just do a simple loop over RepoTags and compare with it ?

  "status": {
    "id": "sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee",
    "repoTags": [
      "k8s.gcr.io/pause:3.6",
      "registry.k8s.io/pause:3.6"
    ],
    "repoDigests": [
      "k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db",
      "registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db"
    ],
    "size": "682696",
    "uid": {
      "value": "65535"
    },
    "username": "",
    "spec": null,
    "pinned": false
  },

  "status": {
    "id": "sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c",
    "repoTags": [
      "registry.k8s.io/pause:3.9"
    ],
    "repoDigests": [
      "registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097"
    ],
    "size": "743952",
    "uid": {
      "value": "65535"
    },
    "username": "",
    "spec": null,
    "pinned": true
  },

[afbjorklund]

The current kubelet flag has been removed in 1.29, along with other missing dockershim leftovers:

kubernetes/kubernetes@d5690f1

So unless the sandbox image is pinned by cri-dockerd, it will now be garbage collected by the kubelet.

kubernetes/kubernetes@d3ae0a3

---

• pkg/kubelet: allow sandbox image pinning from CRI kubernetes/kubernetes#118544
• Clean up dockershim flags in the kubelet kubernetes/kubernetes#106893

[afbjorklund]

Good thing that the "pause" image is normally in use, now when the cleaning by kubelet has started:

REPOSITORY                                TAG               IMAGE ID       CREATED         SIZE
registry.k8s.io/pause                     3.9               e6f181688397   13 months ago   744kB

"Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold":

Nov 19 14:22:45 minikube kubelet[5545]: I1119 14:22:45.466491    5545 image_gc_manager.go:458] "Removing image to free bytes" imageID="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" size=743952 runtimeHandler=""
Nov 19 14:22:45 minikube kubelet[5545]: E1119 14:22:45.468675    5545 remote_image.go:212] "RemoveImage from image service failed" err="rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference \"registry.k8s.io/pause:3.9\" (must force) - container 6fae6463b255 is using its referenced image e6f181688397" image="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
Nov 19 14:22:45 minikube kubelet[5545]: E1119 14:22:45.468713    5545 kuberuntime_image.go:138] "Failed to remove image" err="rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference \"registry.k8s.io/pause:3.9\" (must force) - container 6fae6463b255 is using its referenced image e6f181688397" image="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c"
Nov 19 14:22:45 minikube kubelet[5545]: E1119 14:22:45.468733    5545 kubelet.go:1442] "Image garbage collection failed multiple times in a row" err="wanted to free 47879980646 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference \"registry.k8s.io/pause:3.9\" (must force) - container 6fae6463b255 is using its referenced image e6f181688397"

Server Version: v1.29.0-alpha.3

---

Note: garbage collection is normally turned off in minikube, I enabled it just to check this issue

# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
  nodefs.available: "0%"
  nodefs.inodesFree: "0%"
  imagefs.available: "0%"

[brandond]

Is anyone at Mirantis actively working on this, or is this going to wait for someone in the community to push it across the line? I see that some of this is being worked in #194.

[afbjorklund]

Now included in v0.3.10