[KEP-2400] Node swap updates, GA criterias and clarifications #4701

iholder101 · 2024-06-06T08:27:59Z

One-line PR description:
Add updates, GA criterias and clarifications

Issue link: Node memory swap support #2400

Other comments:

This PR updates the KEP in the following ways:

Emphasize that this KEP is about basic swap enablement
The original KEP indicated that pod-level swap APIs are out of scope:

enhancements/keps/sig-node/2400-node-swap/README.md

Lines 163 to 166 in 155a949

    
           - Allocating swap on a per-workload basis with accounting (e.g. pod-level 
        
             specification of swap). If desired, this should be designed and implemented 
        
             as part of a follow-up KEP. This KEP is a prerequisite for that work. Hence, 
        
             swap will be an overcommitted resource in the context of this KEP.

enhancements/keps/sig-node/2400-node-swap/README.md

Lines 142 to 144 in 155a949

    
           This KEP will be limited in scope to the first two scenarios. The third can be 
        
           addressed in a follow-up KEP. The enablement work that is in scope for this KEP 
        
           will be necessary to implement the third scenario.

However, the lack of APIs and the implicit nature of the current implementation sometimes brings suggestions to extend the API under this KEP.

This KEP focuses on a basic swap enablement. Follow-up KEPs regarding several topics (e.g. customization, zram/zswap suport, and more) will be introduced in the near future, in which we would be able to design and implement each extension in a focused way.

To ensure we're on the same page, this topic was recently raised in a sig-node meeting. In this meeting there was a very broad consensus that this approach makes sense, especially since the NodeSwap feature is important to many different parties which want it to "just work".

This PR updates the KEP to emphasize this approach.

GA criterias
The PR adds GA criterias, alongside the intent to GA in version 1.32.

Make sure PRR is ready

Updates
Since the last KEP updates, many improvements were made and many concerns were addressed. For example:

Memory-backed volumes
Added metrics
Kubelet Configuration examples
more

This PR updates the KEP to reflect these updates.

k8s-ci-robot · 2024-06-06T08:28:09Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: iholder101
Once this PR has been reviewed and has the lgtm label, please assign dchen1107 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

iholder101 · 2024-06-06T08:29:45Z

/cc @SergeyKanzhelev @haircommander @harche

keps/sig-node/2400-node-swap/README.md

deads2k · 2024-06-10T14:24:09Z

Please update https://github.com/kubernetes/enhancements/blob/master/keps/prod-readiness/sig-node/2400.yaml and update missing bits of the PRR questionaire.

keps/sig-node/2400-node-swap/README.md

iholder101 · 2024-06-18T10:49:45Z

Thanks @deads2k!

Please update master/keps/prod-readiness/sig-node/2400.yaml

I see you're the assigned approver for alpha/beta.
Is it OK to also assign you as the approver for GA?

and update missing bits of the PRR questionaire.

Done! PTAL :)

Signed-off-by: Itamar Holder <[email protected]>

…avior Signed-off-by: Itamar Holder <[email protected]>

Signed-off-by: Itamar Holder <[email protected]>

sftim · 2024-06-24T11:24:58Z

/retitle [KEP-2400] Node swap ppdates, GA criterias and clarifications

sftim · 2024-06-24T11:25:21Z

D'oh

/retitle [KEP-2400] Node swap updates, GA criterias and clarifications

sftim

Thanks for the PR.

Here's a mix of feedback; I hope it is all useful.

sftim · 2024-06-24T11:26:25Z

keps/sig-node/2400-node-swap/README.md

@@ -1,4 +1,4 @@
-# KEP-2400: Node system swap support
+# KEP-2400: Node system swap enablement


Suggested change

# KEP-2400: Node system swap enablement

# KEP-2400: Node memory swap support

The authoritative title is the title of the KEP issue.

sftim · 2024-06-24T11:26:48Z

keps/sig-node/2400-node-swap/kep.yaml

Line 1 should read

title: Node memory swap support

The authoritative title is the title of the KEP issue.

sftim · 2024-06-24T11:27:11Z

keps/sig-node/2400-node-swap/README.md

+When `--fail-swap-on=false` is provided to Kubelet but swap is not configured
+otherwise it is guaranteed that, by default, no Kubernetes workloads would
+be able to utilize swap. However, everything outside of kubelet's reach
+(e.g. system daemons, kubelet, etc) would be able to use swap.

 ## Infrastructure Needed (Optional)


Suggested change

## Infrastructure Needed (Optional)

## Infrastructure Needed

(we do need this section)

sftim · 2024-06-24T11:30:30Z

keps/sig-node/2400-node-swap/README.md

@@ -431,6 +457,7 @@ To avoid exhausting swap on a node, `UnlimitedSwap` was dropped from the API in
 #### Security risk

 Enabling swap on a system without encryption poses a security risk, as critical information, such as Kubernetes secrets, may be swapped out to the disk. If an unauthorized individual gains access to the disk, they could potentially obtain these secrets. To mitigate this risk, it is recommended to use encrypted swap. However, handling encrypted swap is not within the scope of kubelet; rather, it is a general OS configuration concern and should be addressed at that level. Nevertheless, it is essential to provide documentation that warns users of this potential issue, ensuring they are aware of the potential security implications and can take appropriate steps to safeguard their system.
+This is now documented and published, e.g. in this blog post: https://kubernetes.io/blog/2023/08/24/swap-linux-beta/.


Suggested change

This is now documented and published, e.g. in this blog post: https://kubernetes.io/blog/2023/08/24/swap-linux-beta/.

The documentation updates are required; there is already a [blog article](https://kubernetes.io/blog/2023/08/24/swap-linux-beta/) that mentions the security implications.

KEPs describe the “desired state”; we mostly don't document progress inline.

Security documentation needs to be done within the guidance itself, such as in https://kubernetes.io/docs/concepts/security/security-checklist/, and not just in blog articles.

sftim · 2024-06-24T11:32:27Z

keps/sig-node/2400-node-swap/README.md

+* If the kernel version equals or is above 6.4, the tmpfs noswap option is being used when necessary.
+* Else, kubelet would try to mount a dummy volume with the tmpfs noswap option to understand whether the option is
+  backported. If the mount succeeds, the tmpfs noswap option is being used when necessary.
+* Else, kubelet would raise a warning about the option not being supported and the possible risk.


Where does the warning appear?

sftim · 2024-06-24T11:34:07Z

keps/sig-node/2400-node-swap/README.md

@@ -561,6 +603,46 @@ type SwapStats struct {
 }
 ```

+In addition, we've added swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and


Suggested change

In addition, we've added swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and

In addition, we'll add swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and

This is framed as a proposal to enhance, and not a changelog. So the proposed changes should be future tense.

sftim · 2024-06-24T11:34:30Z

keps/sig-node/2400-node-swap/README.md

@@ -561,6 +603,46 @@ type SwapStats struct {
 }
 ```

+In addition, we've added swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and
+`/metrics/resource`):
+```shell


nit: not a valid shell script. Use ```console instead.

sftim · 2024-06-24T11:34:38Z

keps/sig-node/2400-node-swap/README.md

+pod_swap_usage_bytes{namespace="kube-system",pod="coredns-8f5847b64-t9gmr"} 0 1687950863144
+```
+
+```shell


nit: not a valid shell script. Use ```console instead.

sftim · 2024-06-24T11:35:46Z

keps/sig-node/2400-node-swap/README.md

+- Nodes with swap enabled -> `node.swap.swapAvailableBytes` should be non-zero.
+- Nodes with memory pressure -> `node.swap.swapUsageBytes` should be non-zero.
+- Containers that reach their memory limit threshold -> `pods[i].containers[i].swap.swapUsageBytes` should be non-zero.
+- Pods with containers that reach their memory limit threshold -> `pods[i].swap.swapUsageBytes` should be non-zero. 


(nit)

Suggested change

- Nodes with swap enabled -> `node.swap.swapAvailableBytes` should be non-zero.

- Nodes with memory pressure -> `node.swap.swapUsageBytes` should be non-zero.

- Containers that reach their memory limit threshold -> `pods[i].containers[i].swap.swapUsageBytes` should be non-zero.

- Pods with containers that reach their memory limit threshold -> `pods[i].swap.swapUsageBytes` should be non-zero.

- Nodes with swap enabled → `node.swap.swapAvailableBytes` should be non-zero.

- Nodes with memory pressure → `node.swap.swapUsageBytes` should be non-zero.

- Containers that reach their memory limit threshold → `pods[i].containers[i].swap.swapUsageBytes` should be non-zero.

- Pods with containers that reach their memory limit threshold → `pods[i].swap.swapUsageBytes` should be non-zero.

sftim · 2024-06-24T11:36:22Z

keps/sig-node/2400-node-swap/kep.yaml

@@ -33,7 +33,7 @@ latest-milestone: "v1.30"
 milestone:
  alpha: "v1.22"
  beta: "v1.30"
-  stable: "TBD"
+  stable: "v1.32"


Suggested change

stable: "v1.32"

#stable: "v1.32"

(AIUI)
We can put in a suggestion, but until we're planning that release, it's not a commitment. If we are planning that release, we should change latest-milestone and stage I think.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/node Categorizes an issue or PR as relevant to SIG Node. labels Jun 6, 2024

k8s-ci-robot requested review from derekwaynecarr and mrunalp June 6, 2024 08:28

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 6, 2024

k8s-ci-robot requested review from haircommander, harche and SergeyKanzhelev June 6, 2024 08:29

iholder101 force-pushed the kep2400/post_beta2 branch from 05552e0 to 4905593 Compare June 6, 2024 08:39

iholder101 mentioned this pull request Jun 6, 2024

Node memory swap support #2400

Open

44 tasks

haircommander reviewed Jun 6, 2024

View reviewed changes

keps/sig-node/2400-node-swap/README.md Outdated Show resolved Hide resolved

haircommander reviewed Jun 6, 2024

View reviewed changes

keps/sig-node/2400-node-swap/README.md Outdated Show resolved Hide resolved

kannon92 reviewed Jun 7, 2024

View reviewed changes

keps/sig-node/2400-node-swap/README.md Outdated Show resolved Hide resolved

keps/sig-node/2400-node-swap/README.md Show resolved Hide resolved

iholder101 force-pushed the kep2400/post_beta2 branch 2 times, most recently from 16c4878 to b3f9708 Compare June 9, 2024 09:17

fabiand reviewed Jun 11, 2024

View reviewed changes

keps/sig-node/2400-node-swap/README.md Outdated Show resolved Hide resolved

iholder101 force-pushed the kep2400/post_beta2 branch from b3f9708 to 443db7f Compare June 18, 2024 10:47

iholder101 force-pushed the kep2400/post_beta2 branch from 443db7f to f4af444 Compare June 18, 2024 10:55

iholder101 added 7 commits June 18, 2024 13:55

Emphesize that KEP-2400 is about basic swap enablement

5f47954

Signed-off-by: Itamar Holder <[email protected]>

Risks and Mitigations - Existing use cases of Swap: update NoSwap beh…

550a158

…avior Signed-off-by: Itamar Holder <[email protected]>

Risks and Mitigations - Security risk: Add blog post as documentation

f1cbd92

Signed-off-by: Itamar Holder <[email protected]>

Risks and Mitigations - Memory-backed volumes

366f364

Signed-off-by: Itamar Holder <[email protected]>

API Changes - KubeConfig addition: LimitedSwap update

5f5a0ca

Signed-off-by: Itamar Holder <[email protected]>

API Changes - Swap Metrics

a6c03b2

Signed-off-by: Itamar Holder <[email protected]>

add e2e test criteria for GA

5798fe1

Signed-off-by: Itamar Holder <[email protected]>

iholder101 added 2 commits June 18, 2024 13:55

GA criterias

a1ece59

Signed-off-by: Itamar Holder <[email protected]>

Complete PRR questionaire

4124649

Signed-off-by: Itamar Holder <[email protected]>

iholder101 force-pushed the kep2400/post_beta2 branch from f4af444 to 4124649 Compare June 18, 2024 10:55

k8s-ci-robot changed the title ~~[KEP-2400] Updates, GA criterias and clarifications~~ [KEP-2400] Node swap ppdates, GA criterias and clarifications Jun 24, 2024

k8s-ci-robot changed the title ~~[KEP-2400] Node swap ppdates, GA criterias and clarifications~~ [KEP-2400] Node swap updates, GA criterias and clarifications Jun 24, 2024

sftim reviewed Jun 24, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KEP-2400] Node swap updates, GA criterias and clarifications #4701

[KEP-2400] Node swap updates, GA criterias and clarifications #4701

iholder101 commented Jun 6, 2024 •

edited

Loading

k8s-ci-robot commented Jun 6, 2024

iholder101 commented Jun 6, 2024

deads2k commented Jun 10, 2024

iholder101 commented Jun 18, 2024

sftim commented Jun 24, 2024

sftim commented Jun 24, 2024

sftim left a comment

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

sftim Jun 24, 2024

	- Allocating swap on a per-workload basis with accounting (e.g. pod-level
	specification of swap). If desired, this should be designed and implemented
	as part of a follow-up KEP. This KEP is a prerequisite for that work. Hence,
	swap will be an overcommitted resource in the context of this KEP.

	This KEP will be limited in scope to the first two scenarios. The third can be
	addressed in a follow-up KEP. The enablement work that is in scope for this KEP
	will be necessary to implement the third scenario.

		@@ -1,4 +1,4 @@
		# KEP-2400: Node system swap support
		# KEP-2400: Node system swap enablement

	# KEP-2400: Node system swap enablement
	# KEP-2400: Node memory swap support

	This is now documented and published, e.g. in this blog post: https://kubernetes.io/blog/2023/08/24/swap-linux-beta/.
	The documentation updates are required; there is already a [blog article](https://kubernetes.io/blog/2023/08/24/swap-linux-beta/) that mentions the security implications.

	In addition, we've added swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and
	In addition, we'll add swap to stats to Summary API and Prometheus endpoints (`/stats/summary` and

[KEP-2400] Node swap updates, GA criterias and clarifications #4701

Are you sure you want to change the base?

[KEP-2400] Node swap updates, GA criterias and clarifications #4701

Conversation

iholder101 commented Jun 6, 2024 • edited Loading

k8s-ci-robot commented Jun 6, 2024

iholder101 commented Jun 6, 2024

deads2k commented Jun 10, 2024

iholder101 commented Jun 18, 2024

sftim commented Jun 24, 2024

sftim commented Jun 24, 2024

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iholder101 commented Jun 6, 2024 •

edited

Loading