Cannot install microk8s 1.29-strict/stable (rev 6373) on ARM (Arch Linux)

[taurus-forever]

Summary

Data team is testing Charmed PostgreSQL K8s for ARM and currently cannot install microk8s from snap (strict version).
Note: non-strict version works well, but Juju 3 requires strict MicroK8s.
Tested with 1.25-strict, 1.27-strict, 1.28-strict and 1.29-strict channels (the same error).

What Should Happen Instead?

microk8s should be installed using snap install command without errors and all components should start for Juju usage.

Reproduction Steps

[root@alarm ~]# snap install microk8s --channel 1.29-strict/stable
error: cannot perform the following tasks:
- Run install hook of "microk8s" snap if present (run hook "install": 
-----
p/microk8s/6373/bin/cat /var/snap/microk8s/6373/certs/kubelet.key
++ /snap/microk8s/6373/usr/bin/base64 -w 0
+ key_data=LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQkFBS0NBUUVBMER5MWV5ckNrMHZiZ3l6M0ptUHNGY0dEcWRPYWdhOWJRZC8zOHhwOSthak91Q1k2CmxVcTdod1RBY1JzSXBtRWtqYjh2MmlDNGpEMjdqWUNEYWJ5K285dEtGdW1ma3RBanIxalJMOG5EMFMzbi9uci8KSTNQeU80NU9BbUlYaTdKcUVBaVd2SVFwdHpYZkxLS2c1cmhiRFg5SkVpZzA5MnF1VnJvNlNTbWhUNDBnWVNkSwphSnp2TXg1ZnpSWVQ5RUpJcThPaHpVbFNwOSt0MmdlMWRDcVBhbFJ6ZVlxb3I2NnpCUkpnVnNLMW1waW1UTFJ6CmdiWUM4YU52Y3h5bmhHaFE5ek1ua3ZRRVBwaTZtdFZjWW9PanFob3FoajlYbW9yY1pGdmFIMWhXdmlSRDFkZzEKSTFIbTU2K1FSREtiZ3VJeS80Zng5ckxYYzJRaGtMRzBNWmx4aXdJREFRQUJBb0lCQVFESGtPYjhXMGFPazZ5VQpVb3VxSnlraTBJcTJQeTFjREFxUlgydnBlQUl6T2l5akgwQVZ3OEZrN0pEaDFONjlWV3dLektlM2pKUlJFdmlqCjYvNEZYL1gwWXp6bUtJL2R5OHZuU0Y5eVJSTjliVmxnOGMrQzBxUGJuODN6U3I3MkNzN0svdkEzU3hJUG0zbCsKOVh6dW9NL2laNUpIN0hDV2pZNzNYeEpzKzVDMUF2alloTy9ZRFBibnUzeC9KcE1CQXV1SGFjZ3VIa3dZTXh0MQpNL2Y0c1ZyVzJReXdDOUtkNUgwOUlXM0QxVndEMFExTGt0MjlsWEFkL0NaeWgzWXBnd3dmdlh5NlVUSHo1UnJ6Cm1TL0ZibTI1TkJ0R1Q3TzV6T0J5NUFMMVpReUZKQUc2blZoaC9NUEVOaDlpOXdtcDV6WW5LWXNWMVUwdzRzaGwKT1BlTFF2NkJBb0dCQU91ZmZXWi9CSHVySnBjTVd0QndHOW9UN3VyZllZcjlzRWhVS25aUWZLejMzRUVvTjNGUgpob3VwZXQ3eDkrcFkzTUdiZ1Y4eWl3V0xZVlNTQmZvaXExZlRqV0didDVOZGxuZ3NydklaZUtnelY2bStvMHo2CmR5VGM0SUJCdENCUkNjTWJwZ2drQ0huNzg0ZG1FVE9MQmE5blhXaWZacWd1aG5BeFQ0bDg2SDg1QW9HQkFPSSsKNnRleWI5d0V0cmNpV0dPNzRtMWdpalZoZ1NmY3k0Rm1vUGVjejF1ZGRuUTJiQVVEM3ZBRVBnVHBBSWdNWGlTTgpFZW1QS2NLWU1mcWFlRHhMc05IL3hkK1FOSmtIeUdMZkptOGQ2U2ZZRFhMeHdqZGpsb2M5SG5ZcHg0MFdaZ2FtCnMxcU5acTd5V0ptc0hnZFpVakdQbjlEWjlCbVhzRXhhNlNSRDA3TGpBb0dCQU9UMUZ3SzVIRjhib2FVWFlVdTQKRktobHhEekJFZVRndlNLYmZxTjNkSjhUUHhpbGZyS0RtTzB2by9EdVNZeVVDdmVySnh5ZFhZL21yNDVoVXFPTQpldGtkL2YxQ05qRmh1S3hRSnpPajRxWDNOaEVBNldnTTNpZVk1aUpiTmpCTTd1S0RoR1BXdDlIcExPeCtGdXJ5CkJYVTdTUDQ0SHFZQWg2eVBqZzA0cmllcEFvR0JBTk80RHlwVW1Dc3IyaFNjdGx4MmlWalR1K0ppWVd2OVNnL0oKQzU5TUJodjB5ZEhNejk4TmxVWEVvb2tUeG1TS05sczhER1FsamUzRllSZmtEOXBVOXA0K0hWeXQ1UnJEMXJqMwpuRTdsamJlMGpqbGZsZUVoZDg4TTJnRzM3VDdSbEI1clIzSFkyeHVpUEZscVlSdDljZFpoVWl3dW1paU04L1V3CktmYSs1bEFEQW9HQkFORlFTcXVBLzd5ZkhEU1hzbHprSnRiNzhQNmt5QTM2NGI5ZEZyYm5ibndlTkFFOUkrdG0KalhBWXdtOVFsay9mRGRwdWszczJVUWhobURacS9VMWl3V2MrTVd1ZHNCTVBVOWNLSTdzQUY3TVE4bDZtanhuagpNRitIMzNmMFNyblpJNDYwK2pRQU8xa3BKY3RuS29OMk5DWHJQVWxzL2lmYThBU1NPa1V5b1lGcAotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
+ config_file=/var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/cp /snap/microk8s/6373/client-x509.config.template /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/CADATA/LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZlZ0F3SUJBZ0lVRzJwYjl0YlBzQWU3aG9BYktaNXZUNGNvM0VBd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0Z6RVZNQk1HQTFVRUF3d01NVEF1TVRVeUxqRTRNeTR4TUI0WERUSTBNREV5T1RBNU1qTXhNRm9YRFRNMApNREV5TmpBNU1qTXhNRm93RnpFVk1CTUdBMVVFQXd3TU1UQXVNVFV5TGpFNE15NHhNSUlCSWpBTkJna3Foa2lHCjl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUEwZ0dsNy80MExubFUwU09TUDN6S1JUb0ZySTJieTFOay9nazIKM0ZQRHE2RkVOSER6YkZsR2VaREpTR2ZQNytWL0hpT1ltb0p3dWVYYzZjNW1iRkxZN21QK05VS2hEYk8xSUgvOQp2OEZ6MnpnU29HeG5JZ0pZb1Y4UHd3TmdTZmxWQmt5bHZCUUlLckR3ZVVZcElvQU5uOVRDN2U4SXlPcHhienUrCk1ZU0NTbE1vaDdPemVFMW1uSHRhTXAweGh4Qk5pMTRsaXY5cW5PSDEyQkIxbjRzaTMyTFJQWjNZekFJZnVhVUYKandIc2d6RDVDcElpYUFFaDhwclNTc2FBUkx1RkpPa1R6NmM3dmUvSUJ2R3pXTDU4NXlnSWllVFlKc3Z4eERvRQphMTNXUVFEenFhbUZuVFB6L3lYYjVJaUYra01Md2pGNExzZE9sanZMUHhLQ0pHRHZyUUlEQVFBQm8xTXdVVEFkCkJnTlZIUTRFRmdRVUhVUGpyYjJGU1FFa2EzVVh5K1JURW8zU1VYd3dId1lEVlIwakJCZ3dGb0FVSFVQanJiMkYKU1FFa2EzVVh5K1JURW8zU1VYd3dEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQwpBUUVBRmZxWUg1eGxFbCtoa2lWUmtoU3hvN2JqSmZlbm1PbDRTSW9IRUZJTUVadFFyRldybmh0QVNPcGVudE5kCnNpQWRmdzBjdjZOOXNxYU4vRXhha3F0S2drMU9TWXZyY2g4NVp5cDlmKzR5QTBxWEhVSGU4amJFSk9ZL1l6bDMKejVQakoxVE5kNlhqYmZQWkUvZGdDb2xtajlpYUlWSEltdERBMHh6RnlrSGVOd0N3L0dLY2RWS3VZWmJraHJNQgpJTlozUDRGRkdFUElwc3VmUFcyR0xLTEdJcjh1aklySTJXeXhEeWM3b24wT1BjMzBVUjNqVWl5T1ZScThJdGl0CmkxSGlGWWx2VzZZUVJWNjkybURraWZRcmJJbXBUTEF1TnorRUZhNnJXRmIyMHc3b3lhNXYrcmEwQld3aHR6U04KblppaEVtSnYzQ3I1L3BkYVA5WlNSSzdCdGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/NAME/system:node:alarm/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/PATHTOCERT/LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lVY1h6ZzA0OEZpZFNQU211OWlCZUg5SitFTWdJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0Z6RVZNQk1HQTFVRUF3d01NVEF1TVRVeUxqRTRNeTR4TUI0WERUSTBNREV5T1RBNU1qTXhNbG9YRFRNMApNREV5TmpBNU1qTXhNbG93TXpFYU1CZ0dBMVVFQXd3UmMzbHpkR1Z0T201dlpHVTZZV3hoY20weEZUQVRCZ05WCkJBb01ESE41YzNSbGJUcHViMlJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUIKQU5BOHRYc3F3cE5MMjRNczl5Wmo3QlhCZzZuVG1vR3ZXMEhmOS9NYWZmbW96cmdtT3BWS3U0Y0V3SEViQ0taaApKSTIvTDlvZ3VJdzl1NDJBZzJtOHZxUGJTaGJwbjVMUUk2OVkwUy9KdzlFdDUvNTYveU56OGp1T1RnSmlGNHV5CmFoQUlscnlFS2JjMTN5eWlvT2E0V3cxL1NSSW9OUGRxcmxhNk9ra3BvVStOSUdFblNtaWM3ek1lWDgwV0UvUkMKU0t2RG9jMUpVcWZmcmRvSHRYUXFqMnBVYzNtS3FLK3Vzd1VTWUZiQ3RacVlwa3kwYzRHMkF2R2piM01jcDRSbwpVUGN6SjVMMEJENll1cHJWWEdLRG82b2FLb1kvVjVxSzNHUmIyaDlZVnI0a1E5WFlOU05SNXVldmtFUXltNExpCk12K0g4ZmF5MTNOa0laQ3h0REdaY1lzQ0F3RUFBYU1hTUJnd0ZnWURWUjBSQkE4d0RZSUZZV3hoY20ySEJBb3EKQU44d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFHbmVQdUNjR3Ezbm00dEUvVkdCZTRmWGtEMHlLOG9ITFlHQwowUmJpanB4SmpLNkt2V3VxaC9NZ3NLK2szS0pjbkcvVWo2NENUdFpGOWdtcVgwWEJTWlFrUkVyd2tRY3V6YXlTCkhlM05aSXpIMFR3L3N3OWhxWktyUFFoOXZyc1E4N0VPNzlwZUNLUFNqOUh3d0VFWk40WHVOblE2OXJhNmRtYjkKR2doSnhqam5ScUxITXpXcTJCL3lNY1dWOVpUM2lHTHJXYVZuL1Ewa3BUMkdwdmtwdWR6VldRMysrWWx3b1pxagpEN0JpZWdXNUNES0hYWGMrL3lLUHBJUE1kNk5ucDlGZW9rbkRzT2hidnBXS3NINGVubXpPYkJLNjhid2MwOXB4Ck1mRlBrcDVWVk5oM0ZZN0tTTzg0N2ZpMDBJTEE0bmF0c1VuRDYwZVNBTzhwNjc4MHd2WT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/PATHTOKEYCERT/LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQkFBS0NBUUVBMER5MWV5ckNrMHZiZ3l6M0ptUHNGY0dEcWRPYWdhOWJRZC8zOHhwOSthak91Q1k2CmxVcTdod1RBY1JzSXBtRWtqYjh2MmlDNGpEMjdqWUNEYWJ5K285dEtGdW1ma3RBanIxalJMOG5EMFMzbi9uci8KSTNQeU80NU9BbUlYaTdKcUVBaVd2SVFwdHpYZkxLS2c1cmhiRFg5SkVpZzA5MnF1VnJvNlNTbWhUNDBnWVNkSwphSnp2TXg1ZnpSWVQ5RUpJcThPaHpVbFNwOSt0MmdlMWRDcVBhbFJ6ZVlxb3I2NnpCUkpnVnNLMW1waW1UTFJ6CmdiWUM4YU52Y3h5bmhHaFE5ek1ua3ZRRVBwaTZtdFZjWW9PanFob3FoajlYbW9yY1pGdmFIMWhXdmlSRDFkZzEKSTFIbTU2K1FSREtiZ3VJeS80Zng5ckxYYzJRaGtMRzBNWmx4aXdJREFRQUJBb0lCQVFESGtPYjhXMGFPazZ5VQpVb3VxSnlraTBJcTJQeTFjREFxUlgydnBlQUl6T2l5akgwQVZ3OEZrN0pEaDFONjlWV3dLektlM2pKUlJFdmlqCjYvNEZYL1gwWXp6bUtJL2R5OHZuU0Y5eVJSTjliVmxnOGMrQzBxUGJuODN6U3I3MkNzN0svdkEzU3hJUG0zbCsKOVh6dW9NL2laNUpIN0hDV2pZNzNYeEpzKzVDMUF2alloTy9ZRFBibnUzeC9KcE1CQXV1SGFjZ3VIa3dZTXh0MQpNL2Y0c1ZyVzJReXdDOUtkNUgwOUlXM0QxVndEMFExTGt0MjlsWEFkL0NaeWgzWXBnd3dmdlh5NlVUSHo1UnJ6Cm1TL0ZibTI1TkJ0R1Q3TzV6T0J5NUFMMVpReUZKQUc2blZoaC9NUEVOaDlpOXdtcDV6WW5LWXNWMVUwdzRzaGwKT1BlTFF2NkJBb0dCQU91ZmZXWi9CSHVySnBjTVd0QndHOW9UN3VyZllZcjlzRWhVS25aUWZLejMzRUVvTjNGUgpob3VwZXQ3eDkrcFkzTUdiZ1Y4eWl3V0xZVlNTQmZvaXExZlRqV0didDVOZGxuZ3NydklaZUtnelY2bStvMHo2CmR5VGM0SUJCdENCUkNjTWJwZ2drQ0huNzg0ZG1FVE9MQmE5blhXaWZacWd1aG5BeFQ0bDg2SDg1QW9HQkFPSSsKNnRleWI5d0V0cmNpV0dPNzRtMWdpalZoZ1NmY3k0Rm1vUGVjejF1ZGRuUTJiQVVEM3ZBRVBnVHBBSWdNWGlTTgpFZW1QS2NLWU1mcWFlRHhMc05IL3hkK1FOSmtIeUdMZkptOGQ2U2ZZRFhMeHdqZGpsb2M5SG5ZcHg0MFdaZ2FtCnMxcU5acTd5V0ptc0hnZFpVakdQbjlEWjlCbVhzRXhhNlNSRDA3TGpBb0dCQU9UMUZ3SzVIRjhib2FVWFlVdTQKRktobHhEekJFZVRndlNLYmZxTjNkSjhUUHhpbGZyS0RtTzB2by9EdVNZeVVDdmVySnh5ZFhZL21yNDVoVXFPTQpldGtkL2YxQ05qRmh1S3hRSnpPajRxWDNOaEVBNldnTTNpZVk1aUpiTmpCTTd1S0RoR1BXdDlIcExPeCtGdXJ5CkJYVTdTUDQ0SHFZQWg2eVBqZzA0cmllcEFvR0JBTk80RHlwVW1Dc3IyaFNjdGx4MmlWalR1K0ppWVd2OVNnL0oKQzU5TUJodjB5ZEhNejk4TmxVWEVvb2tUeG1TS05sczhER1FsamUzRllSZmtEOXBVOXA0K0hWeXQ1UnJEMXJqMwpuRTdsamJlMGpqbGZsZUVoZDg4TTJnRzM3VDdSbEI1clIzSFkyeHVpUEZscVlSdDljZFpoVWl3dW1paU04L1V3CktmYSs1bEFEQW9HQkFORlFTcXVBLzd5ZkhEU1hzbHprSnRiNzhQNmt5QTM2NGI5ZEZyYm5ibndlTkFFOUkrdG0KalhBWXdtOVFsay9mRGRwdWszczJVUWhobURacS9VMWl3V2MrTVd1ZHNCTVBVOWNLSTdzQUY3TVE4bDZtanhuagpNRitIMzNmMFNyblpJNDYwK2pRQU8xa3BKY3RuS29OMk5DWHJQVWxzL2lmYThBU1NPa1V5b1lGcAotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/client-certificate/client-certificate-data/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/client-key/client-key-data/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/127.0.0.1/127.0.0.1/g /var/snap/microk8s/6373/credentials/kubelet.config
+ /snap/microk8s/6373/bin/sed -i s/16443/16443/g /var/snap/microk8s/6373/credentials/kubelet.config
+ cp -r --preserve=mode /snap/microk8s/6373/default-hooks /var/snap/microk8s/common/hooks
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_COMMON}/hooks
+ chmod -R ug+rwX /var/snap/microk8s/6373/credentials/
+ chmod -R o-rwX /var/snap/microk8s/6373/credentials/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_COMMON}/hooks
+ chmod -R ug+rwX /var/snap/microk8s/6373/certs/
+ chmod -R o-rwX /var/snap/microk8s/6373/certs/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_COMMON}/hooks
+ chmod -R ug+rwX /var/snap/microk8s/6373/args/
+ chmod -R o-rwX /var/snap/microk8s/6373/args/
+ for dir in ${SNAP_DATA}/credentials/ ${SNAP_DATA}/certs/ ${SNAP_DATA}/args/ ${SNAP_COMMON}/hooks
+ chmod -R ug+rwX /var/snap/microk8s/common/hooks
+ chmod -R o-rwX /var/snap/microk8s/common/hooks
+ is_strict
+ /snap/microk8s/6373/bin/cat /snap/microk8s/6373/meta/snap.yaml
+ /snap/microk8s/6373/bin/grep confinement
+ /snap/microk8s/6373/bin/grep -q strict
+ return 0
+ snapctl is-connected k8s-kubelet
+ snapctl restart microk8s.daemon-containerd
error: error running snapctl: cannot perform the following tasks:
- Run service command "restart" for services ["daemon-containerd"] of snap "microk8s" (systemctl command [start snap.microk8s.daemon-containerd.service] failed with exit status 1: stderr:
Job for snap.microk8s.daemon-containerd.service failed because the control process exited with error code.
See "systemctl status snap.microk8s.daemon-containerd.service" and "journalctl -xeu snap.microk8s.daemon-containerd.service" for details.)
-----)
[root@alarm ~]#

Introspection Report

[root@alarm ~]# microk8s inspect
-bash: /var/lib/snapd/snap/bin/microk8s: No such file or directory

:-D

Can you suggest a fix?

No, but happy to assist with tracing from here.

Hardware:

PINE64 Quartz64 Model A 8GB:

[root@alarm ~]# lscpu 
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A55
    Model:               0
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r2p0
    CPU(s) scaling MHz:  100%
    CPU max MHz:         1800.0000
    CPU min MHz:         408.0000
    BogoMIPS:            48.00
    Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Not affected
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected
[root@alarm ~]# 

Tnx!

[neoaggelos]

Hi @taurus-forever

Looks like this is caused by containerd failing to start. microk8s inspect cannot help here, since microk8s is not really installed. Can you check the system logs (journalctl) for any reason why containerd might be failing to start?

Such problems might be caused by cgroups, Is cgroup2 enabled on the machine? What's the output of mount -t cgroup2 on the machine?

[taurus-forever]

Tnx for the quick reply!

Our guess is lack of apparmor in kernel (maybe check and warn end-user clearly if apparmor is necessary):

[root@alarm ~]# aa-status
apparmor not present.

P.S. as requested:

[root@alarm ~]# mount -t cgroup2
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
[root@alarm ~]# 

[ktsakalozos]

Do you see anything interesting in systemctl status snap.microk8s.daemon-containerd.service and journalctl -xeu snap.microk8s.daemon-containerd.service ? Could you share the logs for this service?

[taurus-forever]

@ktsakalozos actually yes... deeply hidden:

...
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33063]: Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)
...

The full output:

[root@alarm ~]# systemctl status snap.microk8s.daemon-containerd.service
x snap.microk8s.daemon-containerd.service
     Loaded: not-found (Reason: Unit snap.microk8s.daemon-containerd.service not found.)
     Active: failed (Result: exit-code) since Mon 2024-01-29 09:39:51 UTC; 1h 44min ago
   Main PID: 33014 (code=exited, status=1/FAILURE)
        CPU: 256ms

Jan 29 09:39:51 alarm systemd[1]: Failed to start Service for snap application microk8s.daemon-containerd.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Scheduled restart job, restart counter is at 5.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Start request repeated too quickly.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Failed with result 'exit-code'.
Jan 29 09:39:51 alarm systemd[1]: Failed to start Service for snap application microk8s.daemon-containerd.
[root@alarm ~]# journalctl -xeu snap.microk8s.daemon-containerd.service
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + ARCH=aarch64
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6260/lib:/snap/microk8s/6260/lib/aarch64-linux-gnu:/snap/microk8s/6260/usr/lib/aarc>
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6260/lib:/snap/microk8s/6260/lib/aarch64-linux-gnu:/snap/microk8s/6260/usr/lib/aarch64-lin>
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + export LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6260/lib:/snap/mi>
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + LD_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void:/snap/microk8s/6260/lib:/snap/microk8s/>
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + export XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + XDG_RUNTIME_DIR=/var/snap/microk8s/common/run
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + mkdir -p /var/snap/microk8s/common/run
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + is_strict
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33057]: + grep confinement
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33058]: + grep -q strict
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33056]: + cat /snap/microk8s/6260/meta/snap.yaml
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + return 0
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33014]: + apparmor_parser -r /snap/microk8s/6260/containerd-profile
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33063]: Cache read/write disabled: interface file missing. (Kernel needs AppArmor 2.4 compatibility patch.)
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33063]: Warning: unable to find a suitable fs in /proc/mounts, is it mounted?
Jan 29 09:39:51 alarm microk8s.daemon-containerd[33063]: Use --subdomainfs to override.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- An ExecStart= process belonging to unit snap.microk8s.daemon-containerd.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The unit snap.microk8s.daemon-containerd.service has entered the 'failed' state with result 'exit-code'.
Jan 29 09:39:51 alarm systemd[1]: Failed to start Service for snap application microk8s.daemon-containerd.
-- Subject: A start job for unit snap.microk8s.daemon-containerd.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit snap.microk8s.daemon-containerd.service has finished with a failure.
-- 
-- The job identifier is 10621 and the job result is failed.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Scheduled restart job, restart counter is at 5.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Automatic restarting of the unit snap.microk8s.daemon-containerd.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Start request repeated too quickly.
Jan 29 09:39:51 alarm systemd[1]: snap.microk8s.daemon-containerd.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The unit snap.microk8s.daemon-containerd.service has entered the 'failed' state with result 'exit-code'.
Jan 29 09:39:51 alarm systemd[1]: Failed to start Service for snap application microk8s.daemon-containerd.
-- Subject: A start job for unit snap.microk8s.daemon-containerd.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- A start job for unit snap.microk8s.daemon-containerd.service has finished with a failure.
-- 
-- The job identifier is 10713 and the job result is failed.

[root@alarm ~]# 

Feel free to close this bugreport OR use it to improve UX and inform user about missing apparmor clearly.

Tnx!

[neoaggelos]

I wonder if apparmor missing from the kernel is something we can silently fail on, since multiple things on the snapd will attempt to set profiles. Same goes for containerd itself when it starts container workloads.

We would have to either silently fail on everything (note: snapd handling about this is outside of our control) or fail hard and fast.

I'm also not sure how to properly raise this, maybe adding the apparmor_replace command on the install hook as well? That way the failure will be much easier to locate.

That said, feels like a very niche kernel already, doesn't it? Which kernel version is this? I would expect more Kubernetes related stuff to fail either way, without even talking about CNI/CSI etc

[taurus-forever]

Kernel (fresh Arch from yesterday):

[root@alarm ~]# uname -a
Linux alarm 6.2.10-1-aarch64-ARCH #1 SMP PREEMPT_DYNAMIC Fri Apr  7 10:32:52 MDT 2023 aarch64 GNU/Linux

Hard to recommend something from here.
Snapd works well itself... (at least https://snapcraft.io/hello-world is installed and working well).
Happy to assist if you need to test something here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.