Jetson Nano mender not updating dtb

[tirankas]

Hey Guys,
I now see a problem when updating the Jetson Nano using the mender.
Update process completes successfully, but dtb is not being updated.
I am using the dunfell branch.

nv_boot_control.conf shows this:
TNSPEC 3448-400-0002-F.0-1-0-jetson-nano-esd-imx-cot-mmcblk0p

Checking the bl_update_payload, I see this:
W>=3448-0000-100

=3448-0002-100
W>=3448-0000-101
=3448-0002-101
=3448-0003-400
W<3448-0000-200
<3448-0002-200
W>=3448-0002-100
W<3448-0000-300
<3448-0002-300
=3448-0003-000
W>=3448-0000-300
=3448-0002-300
W>=3448-0003-400
W>=3448-0000-300
=3448-0002-300
=3448-0003-400
W<3448-0000-200

I don't see error with the u-boot-bup-payload deploy process though.

What am I missing?

Thanks

[ichergui]

Hey @tirankas
It's hard to say anything. Are you using Jetson Nano (eMMC version) or Jetson Nano (SDcard version) ?
@madisongh any thoughts ?

[tirankas]

Hi @ichergui ,
Using Jetson Nano (eMMC version).

Thanks

[madisongh]

Are you using the demo distro, and if so, what branch? IIRC, meta-mender-tegra doesn't include bootloader updates for the t210 platforms (e.g., Nano). We have some additions in the demo distro to add that, though.

[tirankas]

Hi @madisongh
I am not using the demo distro directly but I did integrate most of it to our own layer.
I am using the dunfell 32.5.0 branch.
Can you please pinpoint to changes done in the demo distro to allow bootloader update? Maybe I can start by checking some lead there...

Thanks

[madisongh]

Take a look at the bbappend here.

[tirankas]

Hi @madisongh
I actually have this change but I don't see this state script running during the mender update.
I should see mender runs this during update, shouldn't I?

Thanks

[madisongh]

Ah, oops. I pointed you at the dunfell-l4t-r32.5.0 branch copy of that bbappend, but you're right, that still doesn't install the script for tegra210 platforms. That got added to later branches, along with the switchover to using tegra-boot-tools instead of the NVIDIA update engine. I think the important change is here, if you want to try back-porting that to your branch.

[tirankas]

Thanks @madisongh, this makes sense now, will try this out...

[tirankas]

Hi @madisongh
Well, tried this, had also to pull in the tegra-boot-tools-updater to get the tegra-bootloader-update tool.
Mender update completes successfully now but device is not powering up (no output from console too, probably means it doesn't find Uboot).
Was there anything else like uboot location change?

Is there a clean dunfell compatible source tree I can simply build on?

Thanks

[madisongh]

No output on the serial console, even? It would help to know more about why it's not booting.

Was the device already running a build off your dunfell-l4t-r32.5.0 tree, or some other version? If you're trying to use a straight OTA upgrade between BSP versions, more care is needed to ensure compatibility, since the boot partition layout probably changed.

[tirankas]

Now I do see something on the console:

[0000.237] [L4T TegraBoot] (version 00.00.2018.01-l4t-dd84d362)
[0000.242] Processing in cold boot mode Bootloader 2
[0000.247] A02 Bootrom Patch rev = 1023
[0000.251] Power-up reason: software reset
[0000.254] No Battery Present
[0000.257] pmic max77620 reset reason
[0000.260] pmic max77620 NVERC : 0x0
[0000.264] RamCode = 0
[0000.266] Platform has DDR4 type RAM
[0000.269] max77620 disabling SD1 Remote Sense
[0000.274] Setting DDR voltage to 1125mv
[0000.278] Serial Number of Pmic Max77663: 0x329c6
[0000.285] Entering ramdump check
[0000.288] Get RamDumpCarveOut = 0x0
[0000.291] RamDumpCarveOut=0x0, RamDumperFlag=0xe59ff3f8
[0000.296] Last reboot was clean, booting normally!
[0000.301] Sdram initialization is successful
[0000.305] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.311] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.317] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.323] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.329] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.335] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.340] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.346] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.362] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.369] Platform-DebugCarveout: 0
[0000.372] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.378] Non secure mode, and RB not enabled.
[0000.394] Csd NumOfBlocks=0
[0000.632] *** Booting BFS0.
[0000.635] Read PT from (0:3)
[0000.641] Using BFS PT to query partitions
[0000.645] PT: Partition LNX NOT found !
[0001.337] *** Failing over to KFS1.
[0000.286] [L4T TegraBoot] (version 00.00.2018.01-l4t-dd84d362)
[0000.292] Processing in cold boot mode Bootloader 2
[0000.296] A02 Bootrom Patch rev = 1023
[0000.300] Power-up reason: software reset
[0000.304] No Battery Present
[0000.306] pmic max77620 reset reason
[0000.310] pmic max77620 NVERC : 0x0
[0000.313] RamCode = 0
[0000.315] Platform has DDR4 type RAM
[0000.319] max77620 disabling SD1 Remote Sense
[0000.323] Setting DDR voltage to 1125mv
[0000.327] Serial Number of Pmic Max77663: 0x329c6
[0000.334] Entering ramdump check
[0000.337] Get RamDumpCarveOut = 0x0
[0000.341] RamDumpCarveOut=0x0, RamDumperFlag=0xe59ff3f8
[0000.346] Last reboot was clean, booting normally!
[0000.350] Sdram initialization is successful
[0000.354] SecureOs Carveout Base=0x00000000ff800000 Size=0x00800000
[0000.361] Lp0 Carveout Base=0x00000000ff780000 Size=0x00001000
[0000.366] BpmpFw Carveout Base=0x00000000ff700000 Size=0x00080000
[0000.372] GSC1 Carveout Base=0x00000000ff600000 Size=0x00100000
[0000.378] GSC2 Carveout Base=0x00000000ff500000 Size=0x00100000
[0000.384] GSC4 Carveout Base=0x00000000ff400000 Size=0x00100000
[0000.390] GSC5 Carveout Base=0x00000000ff300000 Size=0x00100000
[0000.395] GSC3 Carveout Base=0x000000017f300000 Size=0x00d00000
[0000.412] RamDump Carveout Base=0x00000000ff280000 Size=0x00080000
[0000.418] Platform-DebugCarveout: 0
[0000.421] Nck Carveout Base=0x00000000ff080000 Size=0x00200000
[0000.427] Non secure mode, and RB not enabled.
[0000.443] Csd NumOfBlocks=0
[0000.675] *** Booting BFS1.
[0000.677] Read PT from (0:3)
[0000.683] Using BFS PT to query partitions
[0000.687] PT: Partition LNX NOT found !
[0001.376] *** Failing over to KFS2.

[tirankas]

Also, before the update running this:
fw_printenv -n mender_boot_part
Configuration file wrong or corrupted

[madisongh]

Again:

Was the device already running a build off your dunfell-l4t-r32.5.0 tree, or some other version?

From the messages you mention, it sounds like there is a mismatch between the existing partition layout and what you're trying to update to. And likely a mismatch in the u-boot builds, with different locations for the environment block.

[tirankas]

Was the device already running a build off your dunfell-l4t-r32.5.0 tree, or some other version? --> Yes, already running 32.5.0 dunfell branch

[madisongh]

Hmm. Have you made any other changes besides adding the state script?

[tirankas]

Hi @madisongh
I picked up a bunch of commits you made from here

The device is currently running with an image before this change and the mender I am trying to apy is with these commits.

Does above commits causes uboot change in a way that it cannot be used with an image that doesn't have these changes?

[madisongh]

IIRC, that initial set of commits covered the cboot platforms. There were additional changes needed for t210/u-boot platforms (commit hashes d460cf6, a6254c8) you might need.

[tirankas]

Thanks @madisongh, will verify these fixes...
Btw, aren't all these changes upstream? I don't see these changes in the meta tegra in the master as well.

Thanks

[madisongh]

BTW, if this is your own distro - did you update meta-tegra to the latest on dunfell-l4t-r32.5.0 also? I did have to update tegra-boot-tools to fix an issue with the Nano updates, so if you don't have version 2.5.1 of that package, be sure to update meta-tegra to latest on that branch to pick up that change.

[tirankas]

I probably didn't, picked up the repos as it is via the tegrademo repo. Is this sufficient to only pick up the 2.5.1 of the tegra boot tools package? Less changes, less risk.

[tirankas]

I checked and I do have 2.5.1 of the boot tools.

[tirankas]

Hi @madisongh
Here is the output of that script:
sudo bash -x /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update

• mnt=
• UBOOTVARS='bootcount mender_boot_part mender_boot_part_hex upgrade_available'
• ubvarfile=
• install_fwenv=
• echo 'Installing NVIDIA bootloader update payload'
  Installing NVIDIA bootloader update payload
  ++ fw_printenv -n mender_boot_part
• new_boot_part=1
  ++ mktemp -d -t nvbup.XXXXXX
• mnt=/tmp/nvbup.3LXvXI
• '[' -z /tmp/nvbup.3LXvXI -o '!' -d /tmp/nvbup.3LXvXI ']'
• mount -o ro /dev/mmcblk0p1 /tmp/nvbup.3LXvXI
• '[' '!' -d /tmp/nvbup.3LXvXI/opt/ota_package ']'
• mount --bind /sys /tmp/nvbup.3LXvXI/sys
• mount --bind /proc /tmp/nvbup.3LXvXI/proc
• mount --bind /dev /tmp/nvbup.3LXvXI/dev
• mount -t tmpfs tmpfs /tmp/nvbup.3LXvXI/run
• '[' -e /tmp/nvbup.3LXvXI/usr/share/u-boot/fw_env.config ']'
• chroot /tmp/nvbup.3LXvXI /usr/bin/tegra-bootloader-update --dry-run /opt/ota_package/bl_update_payload
• chroot /tmp/nvbup.3LXvXI /usr/bin/tegra-bootloader-update --dry-run /opt/ota_package/bl_update_payload
• grep -q 'version partitions are corrupted'
• echo 'ERR: cannot perform bootloader update'
  ERR: cannot perform bootloader update
• cleanup
• '[' -n /tmp/nvbup.3LXvXI ']'
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.3LXvXI/sys
• umount /tmp/nvbup.3LXvXI/sys
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.3LXvXI/proc
• umount /tmp/nvbup.3LXvXI/proc
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.3LXvXI/dev
• umount /tmp/nvbup.3LXvXI/dev
• for d in sys proc dev run
• mountpoint -q /tmp/nvbup.3LXvXI/run
• umount /tmp/nvbup.3LXvXI/run
• mountpoint -q /tmp/nvbup.3LXvXI
• umount /tmp/nvbup.3LXvXI
• rmdir /tmp/nvbup.3LXvXI
• exit 1

Thanks

[madisongh]

I've finally got a chance to go back and re-verify Mender updates on the Nano-eMMC platform with the demo distro, and all updates are still working OK across the three dunfell branches.

Looking at the output you posted, I'd say that the updater detected that the version partitions are corrupted, so there is, in fact, a difference in the boot partition layout between the currently running version and the new version. The script will attempt to work around that if it finds a copy of the U-Boot fw_env.config file in /usr/share/u-boot on the new rootfs (performed here in the libubootenv bbappend). From the log output, it looks like that file is not installed in the new rootfs. Can you check on that?

[tirankas]

Thanks @madisongh
What is their strategy here?
If I take the master branch from the community tegra part, what should I merge in to have the latest and greatest? Would I be facing compatibility issues with other layers if I go that path?

[dwalkes]

If I take the master branch from the community tegra part,

Do you mean the master branch of tegra-demo-distro?

Would I be facing compatibility issues with other layers if I go that path?

I think in general for mender L4T upgrades across releases if you don't see the path outlined in https://docs.google.com/spreadsheets/d/1EWG5QZvtWdk74gz48D4xCbegK-bsqMFJ8XOsyGdUUeI/edit#gid=145133208 you are blazing a new trail and your help to document/resolve issues is much appreciated.

[tirankas]

Hi @dwalkes
By the wayz looking at the discussion for the jetpack 4.6 you guys had, I see one of the items is to move to squashfs.
One of the first changes I did when starting with the Jetsons was to move to squashfs. Aside from flash reduction we gained from this (which is significant), squashfs is proven to be very stable in terms of flash robustness. I have been using squashfs with many Qualcomm chipsets (mdm, snapdragons etc) as a FS for the automotive mainly due to the automotive strict flash robustness needs (15 years in field...). Maybe even adding ubi on top of it would be best as another layer of flexibility, ease of maintenance etc...

Btw, you will need to add squashfs support to uboot to get this to work...

[dwalkes]

Thanks @tirankas I've added a cross-link here from https://github.com/OE4T/meta-tegra/wiki/OE4T-Meeting-Notes-2021-08-12

[dwalkes]

Thanks @madisongh and @tirankas for all your work on OE4T/tegra-demo-distro#154

It looks like there's one remaining issue in the thread here at #798 (reply in thread) which is likely specific to @tirankas custom distro if I understand correctly.

It looks like tegra-demo-distro now uses tegra-boot-tools on all branches. I started a wiki page at https://github.com/OE4T/tegra-demo-distro/wiki/Bootloader-tools-use-in-tegra-demo-distro which links back here and references a bootloader change in existing branches, especially r32.4.3, to help highlight to people that they'll be switching bootloader tools if they rebase their branch of tegra-demo-distro, gives an overview of some of the motivation for this. @atharvanan1 can you review when you have a chance and make sure I got this right?

[atharvanan1]

Yeah, I have verified that we are using tegra-boot-tools in all the branches now - in terms of using tegra-boot-tools in mender tools (can be seen in here). I don't see this change anywhere I'm looking, but I might not be looking at correct place anymore.

[madisongh]

Yeah, that change (to package the full payload) is just in the r32.4.3 branch of the demo distro.

Rather than porting that around to the other branches, it might make sense to update the base recipe in meta-tegra to create a separate package containing that payload, then use that package when creating an image for use with the OTA re-flashing sequence.