Jetson Xavier NX Stuck During Flashing both eMMC and NVMe SSD

[wico-mellone]

Hi,

We are experiencing an issue with our NVIDIA Jetson Xavier NX where the board gets stuck during the flashing process.

We have attempted to flash the eMMC and the NVMe on the Jetson Xavier NX using both the NVIDIA development board and our custom board. However, in both cases, the flashing process halts.

Furthermore, the board remains in recovery mode despite the FORCE_RECOVERY* pin being high.
We cannot retrieve serial logs from the target.

Summary:

• Boards Used: Jetson Xavier NX Developer Kit, and our custom board
• Flashing Methods: NVIDIA SDK and custom Yocto build
• Jetpack Version: 5.1.2
• Issue when flashing the eMMC (tried with both NVIDIA SDK and Yocto): The board gets stuck at tegrarcm_v2 --isapplet
• Issue when flashing the NVMe SSD (tried with Yocto): The board gets stuck at == Step 3: Sending flash sequence commands at 2024-06-14T17:18:13+02:00 ==
Waiting for USB storage device flashpkg from 565ffc03...........
• Recovery Mode: The board is always in recovery mode.
• Serial Console: Unable to retrieve any output. To exclude setup issues, we used the same carrier board and the same USB-TTL with another Jetson, and we managed to retrieve the serial logs.

Host-side log from the eMMC attempt flashing with the NVIDIA SDK manager

host_side_log_emmc_flashing_with_nvidia_sdk.log

Host-side log from the NVMe attempt flashing with our custom Yocto build

host_side_log_nvme_flashing_with_yocto.log

Thank you for your help.

[dwalkes]

Serial Console: Unable to retrieve any output. To exclude setup issues, we used the same carrier board and the same USB-TTL with another Jetson, and we managed to retrieve the serial logs.

I think this must mean you aren't on the correct console port somehow? When you mention "another Jetson" you mean a different machine? You should definitely see serial console logs if you get this far, and I'd probably want to start there. Have you tried connecting the UART TXD and UART RXD pins on the Nvidia Developer kit for the Xavier NX?

I just tried with the connection to the NVIDIA dev board shown in these pictures and got these messages before the Step 2: Boot Jetson via RCM at 2024-06-24T18:34:21-06:00 from initrd-flash

[0007.242] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2 from HW fuses.

[0007.250] I> MB1 (prd-version: 2.6.0.0-t194-41334769-cab45716)

[0007.256] I> Boot-mode: Platform RCM

[0007.259] I> Platform: Silicon

[0007.262] I> Chip revision : A02P

[0007.265] I> Bootrom patch version : 15 (correctly patched)

[0007.270] I> ATE fuse revision : 0x200

[0007.274] I> Ram repair fuse : 0x1

[0007.277] I> Ram Code : 0x0

[0007.279] I> rst_source: 0x0, rst_level: 0x0

[0007.284] I> USB configuration success

[0009.496] I> mb2 image downloaded

[0009.574] I> Recovery boot mode 0

[0009.580] I> UPHY full init done

[0009.587] I> MB1 done



[0009.592] I> Welcome to MB2(TBoot-BPMP) Applet (version: default.t194-mobile-dba32e6d)

[0009.599] I> DMA Heap @ [0x40020000 - 0x40065800]

[0009.604] I> Default Heap @ [0xd486400 - 0xd48a400]

[0009.608] W> Profiler not initialized

[0009.714] I> sdmmc DDR50 mode

[0009.721] I> QSPI-0l initialized successfully

[0009.728] E> CLK_RST: instance 6 not found in module 44.

[0009.733] E> MPhy CAR configuration failed error = 1747992077

[0009.738] E> UFS initialization failed

[0009.741] I> UFS is not present

[0009.747] W> Cannot find any partition table for 00000003

[0009.752]  > PARTITION_MANAGER: Failed to publish partition.

[0009.761] I> Found 23 partitions in SDMMC_USER (instance 3)

[0009.767] W> Cannot find any partition table for 00030000

[0009.772]  > PARTITION_MANAGER: Failed to publish partition.

[0009.778] W> Profiler not initialized

[0009.781] I> Entering 3p server

[0009.784] I> USB configuration success

[0010.881] I> Populate chip info

[0010.963] I> Populate eeprom info

[0010.967] I> Populate eeprom info for module cvm

[0010.972]  > DEVICE_PROD: device prod is not initialized.

[0011.179] I> Rebooting : reboot-recovery

Issue when flashing the eMMC

Do you see the same issue with ./doflash.sh as well as ./initrd-flash?

[dwalkes]

Could it be possible that we corrupted the QSPI Flash? I guess that the serial drivers are initialized there, during the booting process at that stage.

Perhaps, I'm not sure.

I was also wondering: Would it be maybe possible to remap the serial console to other Jetson pins?

I don't know about this either, sorry.

Might be a good thing to try on the NVIDIA forum if you haven't already, especially since you can reproduce with stock Jetpack.

[wico-frad]

Hi,

We have raised a ticket on the NVIDIA forum regarding this issue, but unfortunately, there isn't much progress on that front. We are currently suspecting that the memory QSPI might be corrupted. If anyone has encountered a similar problem or has any insights, your help would be greatly appreciated.

Thank you.

[madisongh]

If you're not seeing anything on the serial console at all during the flashing process, then it sounds like you have a bad module. RCM booting only uses the boot ROM, and isn't dependent on the contents of the SPI flash.

[wico-frad]

If you're not seeing anything on the serial console at all during the flashing process, then it sounds like you have a bad module. RCM booting only uses the boot ROM, and isn't dependent on the contents of the SPI flash.

Thank you for your response. To clarify, the module was previously operational, indicating that the hardware was functional at that time. However, we are now encountering an issue where we are unable to flash the eMMC or NVMe SSD using the flashing script provided within the Tegraflash package.

We are currently in the process of diagnosing the root cause of this issue. Our focus is on determining whether the problem is related to hardware failure or another underlying issue. Here is a summary of our observations and steps taken so far:

Initial Module State: The module was verified to be working correctly prior to the current issue.
Current Symptoms: here is no output visible on the serial console during the flashing process.
Flashing Attempts: We have attempted to flash the eMMC or NVMe SSD using different options provided by the TegraFlash script, including --spi-only and --rcm-boot, but these attempts have been unsuccessful.

Please when stating that "RCM booting only uses the boot ROM, and isn't dependent on the contents of the SPI flash," does it mean that the process of RCM (Recovery Mode) booting relies solely on the BootROM embedded within the hardware and then consequently, the serial module (UART) is initialized very early during the boot process, directly by the boot ROM, and should therefore be available regardless of the SPI flash contents.

Any help you can provide would be greatly appreciated.

[madisongh]

Please when stating that [...] the serial module (UART) [...] should therefore be available regardless of the SPI flash contents

Yes, that's what I meant.