kubeadm: Add kubelet instance configuration to configure CRI socket for each node

[HirazawaUi]

• One-line PR description: Add kubelet instance configuration to configure CRI socket for each node.

• Issue link: Add kubelet instance configuration to configure CRI socket for each node #4654

• Other comments:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: HirazawaUi
Once this PR has been reviewed and has the lgtm label, please assign cecilerobertmichon for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/sig-cluster-lifecycle/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[HirazawaUi]

/cc @neolit123 @SataQiu @pacoxu

[neolit123]

@HirazawaUi thanks for starting work on the KEP.

• the formatting is inconsistent. sometimes you use quotes around names sometimes not. some quotes are not closed. sometimes cri is not uppercase. please make everything consistent.
• we shouldn't talk about the --cri-scoket flag, you can instead just the kubeadm CRI socket option, the kubelet CRI socket option, the CRI socket node annotation
• this KEP has two important goals that are not so clear 1) don't write the --container-runtime-endpoint flag in the kubeadm-flags.env file 2) stop writing the annotation on the Node object, please make these clearer in the Goals section
• i think the plan for init/join/upgrade is not very clear across the different kubeadm versions until this feature graduates

feature freeze is in two weeks June 14th
https://github.com/kubernetes/sig-release/tree/master/releases/release-1.31

[neolit123]

how about using a feature gate, and why is that not our first choice?

[HirazawaUi]

I initially thought that this feature would be invisible to ordinary users, and feature gate would increase its complexity (and I'm not sure whether it makes sense to use feature gate on kubeadm), so I implemented it in multiple versions to control version skew and compatibility.

If we use feature gate, then the Design Details will become relatively simple. We only need to implement the feature and use feature gate to control whether it is enabled. There is no need to use multiple versions implemented to consider compatibility and version skew.

Let me think and redesign it based on feature gate.

[neolit123]

if the feature gate is the main approach what alternatives do we have? this section "alternatives" must include one or more alternatives that are not the primary choice of the KEP.

[HirazawaUi]

@neolit123 Thanks, I have fixed according to your suggestion.

[carlory]

#3983 adds support for a drop-in kubelet configuration directory. It allows override the configuration for the Kubelet located at /etc/kubernetes/kubelet.conf.

In this KEP, we may need to consider how to work well with the feature in the context of the kubeadm tool.

cc @neolit123 @HirazawaUi

[neolit123]

#3983 adds support for a drop-in kubelet configuration directory. It allows override the configuration for the Kubelet located at /etc/kubernetes/kubelet.conf.

In this KEP, we may need to consider how to work well with the feature in the context of the kubeadm tool.

cc @neolit123 @HirazawaUi

i prefer to not mix core functionality in kubeadm with 3983, because kubeadm already has the patches mechanism and it's an older feature.

if the user uses also 3983, those files will take precedence.

[neolit123]

this is getting in a better shape.

[neolit123]

if the feature gate is the main approach what alternatives do we have? this section "alternatives" must include one or more alternatives that are not the primary choice of the KEP.

[HirazawaUi]

@neolit123 ready for review.

[HirazawaUi]

ping @neolit123, because the freeze is coming

[neolit123]

@HirazawaUi thank you for the updates.
i'm aware that we are approaching feature freeze but this KEP is not in a clear state.
and i'm the only reviewer.

ideally we should have another approver from the kubeadm OWNER file to also +1/-1 the plan.

[neolit123]

so there is a problem with the KEP currently

feature gates have two states: TRUE and FALSE

but the "Design details" and "Graduation Criteria" sections are organized as we both using and not using a feature gate, which is confusing.

your doc should be organized like this:

• in "Design details" explain the new feature gate
• say in which releases it will be alpha, beta, GA
• explain when it will be enabled by default
• explain what happens when it's enabled and not enabled to the different kubeadm commands

[HirazawaUi]

I made some changes, please check if this is what we expected.

[neolit123]

this section Graduation Criteria must explain what is required for the feature to graduate to Beta and GA . there is no graduation to Alpha.

[HirazawaUi]

i'm aware that we are approaching feature freeze but this KEP is not in a clear state.

Ok, if the current KEP can't be approved before the freeze, I will continue to push it in v1.32.

[HirazawaUi]

@SataQiu @pacoxu Do you have the time and willingness to review this KEP?

[pacoxu]

Overall LGTM.

+1 for this approach. Sorry for the late response.

[pacoxu]

Just a naming question.
We have another file in the same dir named kubeadm-flags.env. Should this file be named with kubeadm prefix?

[HirazawaUi]

I’m not sure if we should follow the same naming convention as kubeadm-flags.env, let’s hear what @neolit123 thinks.

[neolit123]

config.yaml is also kubeadm generated but is not prefixed with "kubeadm" 🤔
because of that i'm leaning towards using "instance-config.yaml", but happy to discuss more and hear more ideas around the naming.

[pacoxu]

After GA, we may not add the old alpha annotation.

• We may deprecate the annotation in Beta?

We can keep the annotation if the node is upgraded from older version, and it can be removed manually if they want then.

[HirazawaUi]

We can keep the annotation if the node is upgraded from older version, and it can be removed manually if they want then.

Yes, this is what we want to do, once the feature gate is enabled, we will not set the CRI socket annotation during kubeadm init/join, nor remove it on upgrade.

So I think it has nothing to do with which phase we are in, because we already have feature gates to help us control version skew.