Mender state script fails at boot loader update "Bad address" #895

vinotion · 2022-02-09T16:03:04Z

vinotion
Feb 9, 2022

Hi!

We've been building our Yocto-based TX2 flash image on top of tegra-demo-distro. Adding layers for custom applications and configuration changes etc. we've tried to keep as much as possible as-is. In our finalization phases, we are testing Mender integration and run into the trouble that boot loader updating seems to fail.

Via meta-mender-tegra/recipes-mender/tegra-state-scripts/tegra-state-scripts_1.0.bb, there's a script called redundant-boot-install-script installed as a Mender state script ArtifactInstall_Leave_80_bl-update (so it is executed after the artifact is installed successfully). When building the whole image (using bitbake demo-image-base), a mender artifact is generated next to the ext4 filesystem and the tegraflash tarball. We are currently testing with this artifact.

This state script at some point calls /usr/bin/tegra-bootloader-update /opt/ota_package/bl_update_payload in a chrooted environment, but this command consistently fails with the error message /opt/ota_package/bl_update_payload: Bad address. This failing script makes Mender rollback the whole update. I've checked, and the bootloader payload file is present -- that's not the issue.

My question is: in what direction should we look trying to find out what is going wrong here? I wasn't able to find any information on this "Bad address" error...what does it mean? We haven't modified anything directly related to the boot loader or Mender. Just rolling our own Mender artifact without the failing script seems like a bad choice, as perhaps at some point we are going to have to update the boot loader as well.

I've also grep'ed the tegra-boot-tools code for "Bad address" but couldn't find it. This makes me think the error is originating from the bootloader payload itself?

Any help is greatly appreciated, thanks in advance!

Best regards,

Kris

Answered by vinotion

Feb 10, 2022

Hi Matt,

Thank you very much for the reply. You were indeed right; the CRC error for the EEPROM read fails. We've been using strace to track down the code in tegra-boot-tools and tegra-eeprom-tool.

Here's the relevant snippet of the bootloader update trace:

$ strace /usr/bin/tegra-bootloader-update /opt/ota_package/bl_update_payload

...program setup...

openat(AT_FDCWD, "/sys/module/tegra_fuse/parameters/tegra_chip_id", O_RDONLY) = 3
read(3, "24\n", 64)                     = 3
close(3)                                = 0
openat(AT_FDCWD, "/proc/cmdline", O_RDONLY) = 3
read(3, "console=ttyS0,115200 console=tty"..., 4095) = 369
close(3)                                = 0
brk(NULL)          …

View full answer

madisongh · 2022-02-09T16:28:35Z

madisongh
Feb 9, 2022
Maintainer

"Bad address" is the error string for the EFAULT errno. The tegra-eeprom library sets that error if the CVM EEPROM contents are invalid (e.g., CRC check fails), so I'm guessing that's the problem here. The updater needs the part number from the EEPROM to construct the TNSPEC used for locating BUP entries to install.

0 replies

vinotion · 2022-02-10T10:53:58Z

vinotion
Feb 10, 2022
Author

Hi Matt,

Thank you very much for the reply. You were indeed right; the CRC error for the EEPROM read fails. We've been using strace to track down the code in tegra-boot-tools and tegra-eeprom-tool.

Here's the relevant snippet of the bootloader update trace:

$ strace /usr/bin/tegra-bootloader-update /opt/ota_package/bl_update_payload

...program setup...

openat(AT_FDCWD, "/sys/module/tegra_fuse/parameters/tegra_chip_id", O_RDONLY) = 3
read(3, "24\n", 64)                     = 3
close(3)                                = 0
openat(AT_FDCWD, "/proc/cmdline", O_RDONLY) = 3
read(3, "console=ttyS0,115200 console=tty"..., 4095) = 369
close(3)                                = 0
brk(NULL)                               = 0x55b2b0d000
brk(0x55b2b2e000)                       = 0x55b2b2e000
mmap(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f55511000
openat(AT_FDCWD, "/sys/module/tegra_fuse/parameters/tegra_prod_mode", O_RDONLY) = 3
read(3, "1\n", 64)                      = 2
close(3)                                = 0
openat(AT_FDCWD, "/sys/module/tegra_fuse/parameters/tegra_chip_id", O_RDONLY) = 3
read(3, "24\n", 64)                     = 3
close(3)                                = 0
openat(AT_FDCWD, "/sys/module/tegra_fuse/parameters/tegra_chip_id", O_RDONLY) = 3
read(3, "24\n", 64)                     = 3
close(3)                                = 0
faccessat(AT_FDCWD, "/sys/bus/i2c/devices/7-0050/eeprom", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/i2c-7", O_RDWR)  = 3
ioctl(3, _IOC(_IOC_NONE, 0x7, 0x6, 0), 0x50) = 0
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 256) = 256
close(3)                                = 0
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x20002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
write(3, "/opt/ota_package/bl_update_paylo"..., 48/opt/ota_package/bl_update_payload: Bad address
) = 48
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

This fails inside eeprom_data_valid.

It is curious that the read data seems to be all zeroes (at least in the start of the read).

In our own software, we also use the i2C reading of the EEPROM to print the board module number etc. for logging. And this doesn't read zeroes, but valid board data. This led us to test the i2c reading with i2cdump:

$ i2cdump 7 0x50

No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-7, address 0x50, mode byte
Continue? [Y/n] Y
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 01 00 0d 00 ee 0c e8 03 42 45 00 00 00 00 00 00    ?.?.????BE......
10: 00 00 00 00 36 39 39 2d 38 33 33 31 30 2d 31 30    ....699-83310-10
20: 30 30 2d 42 30 32 20 45 2e 30 00 00 00 00 00 00    00-B02 E.0......
30: 00 00 a1 db a5 4b 04 00 a2 db a5 4b 04 00 00 00    ..???K?.???K?...
40: 00 00 00 00 a3 db a5 4b 04 00 30 34 32 32 32 31    ....???K?.042221
50: 38 30 33 36 37 38 31 00 00 00 00 00 00 00 00 00    8036781.........
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00    ......NVCB?.M1..
a0: a1 db a5 4b 04 00 a2 db a5 4b 04 00 a3 db a5 4b    ???K?.???K?.???K
b0: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ?...............
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c2    ...............?

Interestingly, after running this i2cdump the tegra-bootloader-update succeeds:

$ /usr/bin/tegra-bootloader-update /opt/ota_package/bl_update_payload

  Processing bpmp-fw_b... [OK]
  Processing sce-fw_b... [OK]
  Processing cpu-bootloader_b... [OK]
  Processing spe-fw_b... [no update needed]
  Processing adsp-fw_b... [OK]
  Processing secure-os_b... [OK]
  Processing BMP_b... [OK]
  Processing mts-bootpack_b... [OK]
  Processing mts-preboot_b... [no update needed]
  Processing sc7_b... [OK]
  Processing bpmp-fw-dtb_b... [OK]
  Processing bootloader-dtb_b... [OK]
  Processing VER_b... [no update needed]
  Processing MB1_BCT_b... [no update needed]
  Processing kernel_b... [OK]
  Processing kernel-dtb_b... [OK]
  Processing mb2_b... [no update needed]
  Processing BCT... [offset=3584,no update needed]...[offset=16384,no update needed]...[offset=0,no update needed]...[OK]
  Processing mb1_b... [no update needed]
  Processing mb1... [no update needed]
Slot 1 marked as active for next boot

Looking at the differences between how i2cdump and our software reads the data from the EEPROM, versus how tegra-bootloader-update does it; there's an extra addressing step involved, so it seems:

$ strace i2cdump 7 0x50

... program setup ...
write(2, "No size specified (using byte-da"..., 43No size specified (using byte-data access)
) = 43
openat(AT_FDCWD, "/dev/i2c/7", O_RDWR)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/i2c-7", O_RDWR)  = 3
ioctl(3, _IOC(_IOC_NONE, 0x7, 0x5, 0), 0x7fc95207d8) = 0
ioctl(3, _IOC(_IOC_NONE, 0x7, 0x3, 0), 0x50) = 0
write(2, "WARNING! This program can confus"..., 75WARNING! This program can confuse your I2C bus, cause data loss and worse!
) = 75
write(2, "I will probe file /dev/i2c-7, ad"..., 54I will probe file /dev/i2c-7, address 0x50, mode byte
) = 54
write(2, "Continue? [Y/n] ", 16Continue? [Y/n] )        = 16
fstat(0, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
brk(NULL)                               = 0x55cadcd000
brk(0x55cadee000)                       = 0x55cadee000
read(0, Y
"Y\n", 1024)                    = 2
fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
write(1, "     0  1  2  3  4  5  6  7  8  "..., 72     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
) = 72
write(1, "00: ", 400: )                     = 4
ioctl(3, _IOC(_IOC_NONE, 0x7, 0x20, 0), 0x7fc95206b8) = 0
... etc ...

In our own software, we also write the read address to the EEPROM first.

1 reply

madisongh Feb 10, 2022
Maintainer

In our own software, we also write the read address to the EEPROM first.

Thanks, yes, that might well be the problem here. I've never run into this issue in my testing, and I think I was assuming that using normal read/write would "just work," as implied by the I2C documentation, but I'll bet that it's worked for me because I don't have anything else on my systems trying to read/write anything on that bus.

I've pushed an update to tegra-eeprom-tool that does that initial address write after opening.

vinotion · 2022-02-10T16:58:47Z

vinotion
Feb 10, 2022
Author

That's great! We're glad it's been fixed.

0 replies

vinotion · 2022-02-10T18:41:28Z

vinotion
Feb 10, 2022
Author

FWIW: Here's a working bitbake recipe for v1.3.0:

DESCRIPTION = "Tegra ID EEPROM tools"
HOMEPAGE = "https://github.com/OE4T/tegra-eeprom-tool"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=22c98979a7dd9812d2455ff5dbc88771"

DEPENDS = "libedit"

SRC_URI = "https://github.com/OE4T/${BPN}/releases/download/v${PV}/${BP}.tar.gz"
SRC_URI[sha256sum] = "d240e71d088532694770dab6f13371ebb0f439efdf4fa10b5194e2011dc6177a"

inherit cmake

RRECOMMENDS_${PN} += "kernel-module-at24"

PACKAGES =+ "${PN}-boardspec"
FILES_${PN}-boardspec = "${bindir}/tegra-boardspec"

0 replies

ichergui · 2022-02-10T18:52:53Z

ichergui
Feb 10, 2022
Collaborator

Hey @vinotion

Could you please push a PR ? otherwise @madisongh or myself will update the recipe.
I'm not sure if we will use cmake or meson

Thanks

3 replies

madisongh Feb 10, 2022
Maintainer

I'll be adding the recipe to the PR I already have in the works.

@vinotion thanks for confirming that it worked

vinotion Feb 10, 2022
Author

@ichergui Yes, I considered doing that but wasn't sure if it wasn't already under way. Will do next time!

madisongh Feb 11, 2022
Maintainer

I've updated all of the current meta-tegra branches to v1.3.0 with the fix.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mender state script fails at boot loader update "Bad address" #895

{{title}}

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Mender state script fails at boot loader update "Bad address" #895

vinotion Feb 9, 2022

Replies: 5 comments · 4 replies

madisongh Feb 9, 2022 Maintainer

vinotion Feb 10, 2022 Author

madisongh Feb 10, 2022 Maintainer

vinotion Feb 10, 2022 Author

vinotion Feb 10, 2022 Author

ichergui Feb 10, 2022 Collaborator

madisongh Feb 10, 2022 Maintainer

vinotion Feb 10, 2022 Author

madisongh Feb 11, 2022 Maintainer

vinotion
Feb 9, 2022

Replies: 5 comments 4 replies

madisongh
Feb 9, 2022
Maintainer

vinotion
Feb 10, 2022
Author

madisongh Feb 10, 2022
Maintainer

vinotion
Feb 10, 2022
Author

vinotion
Feb 10, 2022
Author

ichergui
Feb 10, 2022
Collaborator

madisongh Feb 10, 2022
Maintainer

vinotion Feb 10, 2022
Author

madisongh Feb 11, 2022
Maintainer