sandbox/cgroup: improve cgroup-based process termination algorithm #14513

ZeyadYasser · 2024-09-16T16:28:20Z

unify termination algorithm for v1/v2

for each snap cgroup:
- while cgroup.procs is not empty:
  - SIGKILL each pid in cgroup.procs
for v1 only, also kill pids found in freezer cgroup created by snap-confine
- this is relevant for systemd v237 (used in ubuntu 18.04) for non-root users where the transient scope cgroups are not created

This logic drops the freeze/kill/thaw approach with all the weird v1/v2/kernel backward compatibility.

This PR should also fix flakey results seen in tests/main/snap-remove-terminate.

ZeyadYasser · 2024-09-16T20:24:59Z

Something is going wrong in the new approach, need to investigate why cgroup.procs reports no such device instead of file does not exist.

+ snap remove --terminate test-snapd-sh
error: cannot perform the following tasks:
- Kill running snap "test-snapd-sh" apps (read /sys/fs/cgroup/user.slice/user-0.slice/[email protected]/app.slice/snap.test-snapd-sh.sh-0e68ddf6-d14f-4c3c-9359-94bf510a9b8d.scope/cgroup.procs: no such device)

ZeyadYasser · 2024-09-17T18:48:55Z

Something is going wrong in the new approach, need to investigate why cgroup.procs reports no such device instead of file does not exist.
+ snap remove --terminate test-snapd-sh
error: cannot perform the following tasks:
- Kill running snap "test-snapd-sh" apps (read /sys/fs/cgroup/user.slice/user-0.slice/[email protected]/app.slice/snap.test-snapd-sh.sh-0e68ddf6-d14f-4c3c-9359-94bf510a9b8d.scope/cgroup.procs: no such device)

Found root cause of ENODEV:

https://github.com/torvalds/linux/blob/a940d9a43e623d1ba1e5c499aa843516656c0ae4/fs/kernfs/file.c#L592-L603 kernfs->kernfs_fop_open
https://github.com/torvalds/linux/blob/a940d9a43e623d1ba1e5c499aa843516656c0ae4/fs/kernfs/file.c#L145-L156 kernfs->kernfs_seq_start

It is kernfs file operations behavior where it checks that the file is still there before continuing. It should be safe to skip errors for ENODEV similar to skipping on fs.ErrNotExist.

What triggers ENODEV is starting opening/reading cgroup.procs and have systemd remove the cgroup at the same time while the kernel operation is ongoing.

bboozzoo · 2024-09-18T12:37:50Z

sandbox/cgroup/kill.go

-//
-//  1. Cgroup v2 freezer was only available since Linux 5.2 so freezing is a no-op before 5.2 which allows processes to keep forking.
-//  2. Freezing does not put processes in an uninterruptable sleep unlike v1, so they can be killed externally and have their pid reused.
-//  3. `cgroup.kill` was introduced in Linux 5.14 and solves the above issues as it kills the cgroup processes atomically.
 func killSnapProcessesImplV2(ctx context.Context, snapName string) error {
 	killCgroupProcs := func(dir string) error {
 		// Use cgroup.kill if it exists (requires linux 5.14+)
 		err := writeExistingFile(filepath.Join(dir, "cgroup.kill"), []byte("1"))
 		if err == nil || !errors.Is(err, fs.ErrNotExist) {


given ENODEV appearing in certain scenarios, I wonder if it is possible for it to show up here as well. When looking at the kernel, could ENODEV be returned for every cgroup meta-file?

Good point, I just checked and weirdly enough, No.

https://elixir.bootlin.com/linux/v6.11/source/kernel/cgroup/cgroup.c#L4030, but I think we should a check for good measure if kernel behavior decided to change later.

It is surprising to see ENOENT/ENODEV thrown everywhere like that without distinction in the kernel source code.

bboozzoo · 2024-09-18T12:42:09Z

sandbox/cgroup/kill.go

-// pids read earlier.
+// Note: When cgroup v1 is detected, the call will also act on the freezer
+// group created when a snap process was started to address a known bug on
+// systemd v327 for non-root users.


maybe worth mentioning that this is only useful for killing apps or processes which do not have their lifecycle managed by external entities like systemd

bboozzoo · 2024-09-18T12:46:51Z

sandbox/cgroup/kill.go

+	// Keep sending SIGKILL signals until no more pids are left in cgroup
+	// to cover the case where a process forks before we kill it.
+	for {
+		// XXX: Should this have maximum retries?


perhaps we could have a spread test which essential does a fork bomb inside the snap, and with this code we should still arrive at a stable state when the cgorup is empty

Thanks, Great idea. I added a fork-bomb test variant. I crashed my machine twice making this 😅

bboozzoo · 2024-09-19T07:38:00Z

needs a rebase now

tests/lib/snaps/fork-bomb/meta/snap.yaml

bboozzoo · 2024-09-19T07:40:09Z

tests/lib/snaps/fork-bomb/meta/snap.yaml

+license: GPL-3.0
+apps:
+ fork-bomb:
+  command: bin/bomb


you can add another app called sh with a trivial script which does exec and then there'd be no need to install test-snapd-sh (which also pulls in core IIRC)

Nice, the test is leaner now. Thanks!

unify termination algorithm for v1/v2 - for each snap cgroup: - while cgroup.procs is not empty: - SIGKILL each pid in cgroup.procs - for v1 only, also kill pids found in freezer cgroup created by snap-confine - this is relevant for systemd v237 (used in ubuntu 18.04) for non-root users where the transient scope cgroups are not created This logic drops the freeze/kill/thaw approach with all the weird v1/v2/kernel backward compatibility. Signed-off-by: Zeyad Gouda <[email protected]>

Signed-off-by: Zeyad Gouda <[email protected]>

This test variant stress-tests the new algorithm where snapd could be racing after a fork bomb without doing freezing first by continuously killing pids that show up until all pids are drained from cgroup. Signed-off-by: Zeyad Gouda <[email protected]>

Signed-off-by: Zeyad Gouda <[email protected]>

bboozzoo

LGTM

bboozzoo · 2024-09-19T12:38:15Z

tests/main/snap-remove-terminate/task.yaml

+    sh_snap_bin="$(command -v fork-bomb.sh)"
+    if [ "$BAD_SNAP" = "fork-bomb" ]; then
+        #shellcheck disable=SC2016
+        systemd-run --unit test-kill.service flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; $SNAP/bin/fork-bomb'


so that it doesn't blow up too much in the tests:

Suggested change

systemd-run --unit test-kill.service flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; $SNAP/bin/fork-bomb'

systemd-run --unit test-kill.service -p TasksMax=1000 flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; $SNAP/bin/fork-bomb'

It seems to only apply to the outer systemd-run, but snapd's systemd-run escapes the outer cgroup limits imposed by TasksMax.

Interestingly, I think we accidentally just discovered that snaps don't behave normally when run as part of systemd directly because snap run calls systemd-run which create a separate unit/cgroup.

andrewphelpsj

Thanks! Some suggestions (and one open question), but looks good.

andrewphelpsj · 2024-09-19T13:23:32Z

sandbox/cgroup/kill.go

 			return err
 		}
 		return nil
 	}

 	var firstErr error
 	skipError := func(err error) bool {
-		if !errors.Is(err, fs.ErrNotExist) && firstErr == nil {
+		// fs.ErrNotExist and ENODEV are ignored in case the cgroup went away while we were
+		// processing the cgorup. ENODEV is returned by the kernel if the cgroup went


Suggested change

// processing the cgorup. ENODEV is returned by the kernel if the cgroup went

// processing the cgroup. ENODEV is returned by the kernel if the cgroup went

good catch, thank you!

andrewphelpsj · 2024-09-19T13:28:47Z

tests/main/snap-remove-terminate/task.yaml

+    else
+        #shellcheck disable=SC2016
+        systemd-run --unit test-kill.service flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; sleep 100000'
+    fi


Does this other variant have much value now?

The way I see it, the normal variant tests that the feature works in the normal case and the other fork-bomb variant stresses a specific part of the code responsible for handling such processes. I like how the fork-bomb variant isolates where to look in case of a regression.

andrewphelpsj · 2024-09-19T13:51:38Z

sandbox/cgroup/kill.go

+		// fs.ErrNotExist and ENODEV are ignored in case the cgroup went away while we were
+		// processing the cgorup. ENODEV is returned by the kernel if the cgroup went
+		// away while a kernfs operation is ongoing.
+		if !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENODEV) && firstErr == nil {


From the open man page:

ENODEV pathname refers to a device special file and no corresponding device exists. (This is a Linux kernel bug; in this situation ENXIO must be returned.) ENXIO The file is a device special file and no corresponding device exists.

I wonder if we should handle ENXIO too?

I think it's ok as is. The error handling branch is tailored to handle the error cases for which the overall operation of killing processes can be considered to have succeeded. Specifically ENOENT and then to account for the cgroup v1 prededent ENODEV. I do not see any evidence of ENXIO use under kernel/cgroup in the kernel tree, so I think it's ok for the code to fail explicitly should it even occur.

Signed-off-by: Zeyad Gouda <[email protected]>

codecov · 2024-09-19T19:20:50Z

Codecov Report

Attention: Patch coverage is 90.47619% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.85%. Comparing base (04d0ab0) to head (c9ef741).
Report is 19 commits behind head on master.

Files with missing lines	Patch %	Lines
sandbox/cgroup/kill.go	90.47%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #14513      +/-   ##
==========================================
+ Coverage   78.83%   78.85%   +0.02%     
==========================================
  Files        1078     1079       +1     
  Lines      145096   145467     +371     
==========================================
+ Hits       114389   114714     +325     
- Misses      23546    23583      +37     
- Partials     7161     7170       +9

Flag	Coverage Δ
unittests	`78.85% <90.47%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ZeyadYasser requested review from zyga and andrewphelpsj September 16, 2024 16:28

ZeyadYasser added this to the 2.66 milestone Sep 16, 2024

ZeyadYasser requested a review from bboozzoo September 16, 2024 16:29

ZeyadYasser added the ⛔ Blocked label Sep 17, 2024

ZeyadYasser force-pushed the improve-cgroup-based-process-termination branch from c6a5813 to 371e0eb Compare September 17, 2024 18:44

ZeyadYasser removed the ⛔ Blocked label Sep 17, 2024

bboozzoo reviewed Sep 18, 2024

View reviewed changes

ZeyadYasser requested a review from bboozzoo September 18, 2024 20:44

bboozzoo reviewed Sep 19, 2024

View reviewed changes

ZeyadYasser added 4 commits September 19, 2024 14:36

sandbox/cgroup: address review comments

1e8cdd6

Signed-off-by: Zeyad Gouda <[email protected]>

tests: address review comments

55b74c3

Signed-off-by: Zeyad Gouda <[email protected]>

ZeyadYasser force-pushed the improve-cgroup-based-process-termination branch from 36bb459 to 55b74c3 Compare September 19, 2024 12:00

ZeyadYasser requested a review from bboozzoo September 19, 2024 12:34

bboozzoo approved these changes Sep 19, 2024

View reviewed changes

andrewphelpsj approved these changes Sep 19, 2024

View reviewed changes

ZeyadYasser added 2 commits September 19, 2024 18:14

many: address review comments

33e92f7

Signed-off-by: Zeyad Gouda <[email protected]>

tests: fix wrong binary path

c9ef741

Signed-off-by: Zeyad Gouda <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sandbox/cgroup: improve cgroup-based process termination algorithm #14513

sandbox/cgroup: improve cgroup-based process termination algorithm #14513

ZeyadYasser commented Sep 16, 2024 •

edited

Loading

ZeyadYasser commented Sep 16, 2024

ZeyadYasser commented Sep 17, 2024

bboozzoo Sep 18, 2024

ZeyadYasser Sep 18, 2024

bboozzoo Sep 18, 2024

bboozzoo Sep 18, 2024

ZeyadYasser Sep 18, 2024

bboozzoo commented Sep 19, 2024

bboozzoo Sep 19, 2024

ZeyadYasser Sep 19, 2024

bboozzoo left a comment

bboozzoo Sep 19, 2024

ZeyadYasser Sep 19, 2024

andrewphelpsj left a comment

andrewphelpsj Sep 19, 2024

ZeyadYasser Sep 19, 2024

andrewphelpsj Sep 19, 2024

ZeyadYasser Sep 19, 2024

andrewphelpsj Sep 19, 2024

bboozzoo Sep 19, 2024

codecov bot commented Sep 19, 2024

	systemd-run --unit test-kill.service flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; $SNAP/bin/fork-bomb'
	systemd-run --unit test-kill.service -p TasksMax=1000 flock "$lockfile" "$sh_snap_bin" -c 'touch /var/snap/test-snapd-sh/common/alive; $SNAP/bin/fork-bomb'

	// processing the cgorup. ENODEV is returned by the kernel if the cgroup went
	// processing the cgroup. ENODEV is returned by the kernel if the cgroup went

sandbox/cgroup: improve cgroup-based process termination algorithm #14513

Are you sure you want to change the base?

sandbox/cgroup: improve cgroup-based process termination algorithm #14513

Conversation

ZeyadYasser commented Sep 16, 2024 • edited Loading

ZeyadYasser commented Sep 16, 2024

ZeyadYasser commented Sep 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bboozzoo commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bboozzoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewphelpsj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 19, 2024

Codecov Report

ZeyadYasser commented Sep 16, 2024 •

edited

Loading