r36.3.0 timeframe

[tbottomparallel]

Hi @madisongh,

Do you have an estimate for when r36.3.0 will be ready for use?

I have run into an issue upgrading from kirkstone r35.4.1 -> r35.5.0. It is not entirely clear if it is a kernel issue or an nvidia issue, but I suspect it is a little bit of both - that nvidia code is running into a corner case with cgroups in the 5.10.192 kernel.

I can reproduce the issue with the current head of the kirkstone branch of tegra-demo-distro but not with wip-l4t-r36.3.0. I spent more time than I should have already digging around kernel scheduler/cgroup code debugging it, but that's not my area of expertise. I'm hoping to skip over the problematic version rather than continue banging my head on it. Alas, time and forehead durability are both finite resources.

---

Not an issue for meta-tegra, but included for your curiosity: the issue itself is trivial to reproduce. Enable the cpuset cgroup controller, create a new cgroup, give it a cpu, and make it a root partition:

mount cgroupfs -t cgroup2 /sys/fs/cgroup/
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/tstgrp
echo +cpuset > /sys/fs/cgroup/tstgrp/cgroup.subtree_control
echo 11 > /sys/fs/cgroup/tstgrp/cpuset.cpus
echo root > /sys/fs/cgroup/tstgrp/cpuset.cpus.partition

which results in a workqueue being run from the wrong cpu:

[  617.767508] ------------[ cut here ]------------
[  617.767683] WARNING: CPU: 1 PID: 119 at kernel/workqueue.c:2197 process_one_work+0x6c/0x4d0
[  617.767921] Modules linked in: nfc bnep nvidia_drm(O) nvidia_modeset(O) snd_hda_codec_hdmi rtl8822ce snd_hda_tegra nvgpu snd_hda_codec cfg80211 pwm_fan rtk_btusb snd_hda_core ina3221 tegra_bpmp_thermal userspace_alert spi_tegra114 nvmap nvidia(O)
[  617.768600] CPU: 1 PID: 119 Comm: kworker/0:4 Tainted: G           O      5.10.192-l4t-35.5.0+gdb857ecdbe10 #1
[  617.768894] Hardware name: Jetson AGX Orin Industrial (DT)
[  617.769062] Workqueue:  0x0 (events)
[  617.769175] pstate: 20c00089 (nzCv daIf +PAN +UAO -TCO BTYPE=--)
[  617.769339] pc : process_one_work+0x6c/0x4d0
[  617.769461] lr : process_one_work+0x28/0x4d0
[  617.769583] sp : ffff8000116ebda0
[  617.769677] x29: ffff8000116ebda0 x28: 0000000000000000
[  617.770407] x27: ffff19a78137b9c8 x26: ffffd493c4ccc878
[  617.771222] x25: ffffd493c49d2420 x24: ffffd493c49b6000
[  617.772038] x23: ffff19b4acf92880 x22: 0000000000000000
[  617.772856] x21: ffff19b4acf96600 x20: ffff19a781581600
[  617.775668] x19: ffff19a78140e080 x18: 0000000000000000
[  617.781093] x17: 0000000000000000 x16: 0000000000000000
[  617.786604] x15: 0000000000000000 x14: ffff19a780161d80
[  617.791943] x13: ffff4520e88dd000 x12: 000000003464d91d
[  617.797368] x11: 0000000000000000 x10: 0000000000000ab0
[  617.802881] x9 : ffff8000116ebd20 x8 : ffff19a7815d6390
[  617.808218] x7 : ffffd493c44d5000 x6 : 00000004ba23c1c3
[  617.813555] x5 : 00000000410fd420 x4 : ffff19b4acf92880
[  617.818980] x3 : ffff19a7815d5880 x2 : ffff4520e88dd000
[  617.824320] x1 : 0000000000000000 x0 : 0000000000000001
[  617.829657] Call trace:
[  617.832108]  process_one_work+0x6c/0x4d0
[  617.835872]  worker_thread+0x144/0x490
[  617.839809]  kthread+0x160/0x164
[  617.843049]  ret_from_fork+0x10/0x18
[  617.846545] ---[ end trace c289d81a6d0e61fb ]---
[  629.220070] run queue from wrong CPU 2, hctx active

If an nvme drive is installed and mounted it will also periodically print the following every 5-6 seconds:

[  629.220247] CPU: 2 PID: 6 Comm: kworker/0:0H Tainted: G        W  O      5.10.192-l4t-35.5.0+gdb857ecdbe10 #1
[  629.220536] Hardware name: Jetson AGX Orin Industrial (DT)
[  629.220702] Workqueue: kblockd blk_mq_run_work_fn
[  629.220840] Call trace:
[  629.220930]  dump_backtrace+0x0/0x1bc
[  629.221045]  show_stack+0x30/0x40
[  629.221148]  dump_stack+0xe8/0x120
[  629.221250]  __blk_mq_run_hw_queue+0x128/0x130
[  629.221385]  blk_mq_run_work_fn+0x2c/0x40
[  629.221508]  process_one_work+0x1d8/0x4d0
[  629.221626]  worker_thread+0x144/0x490
[  629.221741]  kthread+0x160/0x164
[  629.221827]  ret_from_fork+0x10/0x18
[  635.363835] run queue from wrong CPU 2, hctx active

[madisongh]

We have JetPack 6/L4T R36.3.0 in the master and scarthgap branches. We have a page on the wiki with the branch and version map.

[tbottomparallel]

perfect, thank you!