Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to IDF 5.3.1 #3664

Open
wants to merge 6 commits into
base: dev-esp32
Choose a base branch
from
Open

Conversation

jmattsson
Copy link
Member

  • This PR is for the dev-esp32 branch rather than for the release branch.
  • This PR is compliant with the other contributing guidelines as well (if not, please describe why).
  • I have thoroughly tested my contribution.
  • [n/a] The code changes are reflected in the documentation at docs/*.

The core of this PR is the upgrade of the ESP IDF to the latest, 5.3.1. There were only a few adjustments required to make NodeMCU build with it, and a couple more to make it run. Specifically, the VFS console driver needed to be explicitly initialised now, and our existing issue with WiFi+USB-CDC had a regression which necessitated a change to the workaround. The workaround is only enabled if USB-CDC console is used.

A second issue was discovered on the ESP32-S2 with USB-CDC enabled, whereby flashing of a blank unit failed to start up. After quite a bit of digging I narrowed this down to the auto-format of the SPIFFS partition. That much flash access seems to interfere with the USB device enumeration, and results in a failed boot / the unit being repeatedly reset by the USB host. When the USB-CDC console is enabled, we now default to not auto-formatting, though this can be overridden via the new Kconfig option.

While poking around in this area I also discovered another pre-existing issue where if the initial SPIFFS mount failed, then a file.format() would not mount the newly formatted filesystem, necessitating a node.restart() before it was usable. This has now been addressed by requesting a (re)mount after a successful format, giving consistent behaviour in both cases. As part of fixing this and the previous issue, the default filesystem mounting code has been moved into our platform component, as it's needed both by user_main.c and the file module.

Basic testing has been done on: ESP32, ESP32-C3, ESP32-C6, ESP32-H2 and ESP32-S2.
Additional testing by others is welcome and encouraged.

jmattsson and others added 4 commits October 14, 2024 11:57
- If the initial filesystem mounting fails, then a subsequent file.format()
  does not mount it afterwards, necessitating a node.restart(). In the normal
  case, after file.format() the filesystem is mounted and ready for use.
  We now explicitly remount after a format to gain consistent behaviour.

- If using USB CDC, the auto-format on first boot interferes with USB
  enumeration, resulting in a failed startup. We now avoid doing an
  auto-format if the USB CDC console is enabled. The behaviour is
  configurable/overrideable via Kconfig.
@serg3295
Copy link

I have tested the PR on esp32, esp32-c3 super mini, nanoESP32-C6 v1.0, esp32-s3 zero, esp32-s2 Mini v1.0.0 for correct operation with nodemcu-tool utility and nodemcu-tools VSCode extension.

all connections via USB

esp32
port: UART0 - works as expected.

esp32c3 and esp32c6
port: usb-serial-jtag
behavior =>
uart.write(0) causes 'uart driver error' message
using
uart.start(0) -- no 'driver error' message, but
uart.write(0,'a\n') -- nothing is sent to the console

Applying changes as described in issue #3657 solves the problem. Both utilities work fine.

nano esp32-c6
port: UART0 (ch343) -- works as expected without changing "platform.c"

esp32-s3 zero
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded Flash 4MB (XMC), Embedded PSRAM 2MB (AP_3v3)

port: usb-serial-jtag
idf5.1.3 REPL -> O.K.
idf5.2.2 REPL -> O.K.
idf5.2.3 REPL, but typing any char in console reboots esp
idf5.3.1 repeated reboot

esp32-s2 Mini
Chip is ESP32-S2FNR2 (revision v0.0)
Features: WiFi, Embedded Flash 4MB, Embedded PSRAM 2MB, ADC and temperature sensor calibration in BLK2 of efuse V2
port: USB-CDC
behavior =>
I have only REPL in console (make monitor /dev/ttyACM0). Both utilities does not work.
Sending simple print("echo1337") sometimes causes

E (295) mmap: esp_mmu_paddr_to_vaddr(805): paddr isn't mIprint("echo1337")

Wifi errors

the same issue #3660 is on esp32c3 and esp32c6 with idf5.3.1

@serg3295
Copy link

Fixes warnings

diff --git a/components/base_nodemcu/user_main.c b/components/base_nodemcu/user_main.c
index 82453afd..f2391b22 100644
--- a/components/base_nodemcu/user_main.c
+++ b/components/base_nodemcu/user_main.c
@@ -24,6 +24,7 @@
 #include "esp_vfs_usb_serial_jtag.h"
 #include "driver/uart_vfs.h"
 #include "driver/usb_serial_jtag.h"
+#include "driver/usb_serial_jtag_vfs.h"
 #include "nvs_flash.h"
 
 #include "task/task.h"
@@ -240,15 +241,15 @@ static void console_init(void)
 #elif CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG
   /* Based on @pjsg's work */
 
-  esp_vfs_dev_usb_serial_jtag_set_rx_line_endings(RX_LINE_ENDINGS_CFG);
-  esp_vfs_dev_usb_serial_jtag_set_tx_line_endings(TX_LINE_ENDINGS_CFG);
+  usb_serial_jtag_vfs_set_rx_line_endings(RX_LINE_ENDINGS_CFG);
+  usb_serial_jtag_vfs_set_tx_line_endings(TX_LINE_ENDINGS_CFG);
 
   usb_serial_jtag_driver_config_t usb_serial_jtag_config =
     USB_SERIAL_JTAG_DRIVER_CONFIG_DEFAULT();
   /* Install USB-SERIAL-JTAG driver for interrupt-driven reads and write */
   usb_serial_jtag_driver_install(&usb_serial_jtag_config);
 
-  esp_vfs_usb_serial_jtag_use_driver();
+  usb_serial_jtag_vfs_use_driver();
 #elif CONFIG_ESP_CONSOLE_USB_CDC
   /* Based on console/advanced_usb_cdc */
 

@jmattsson
Copy link
Member Author

@serg3295 Oooh, thank you for testing!

Let's see what we can do about these (and what I can reproduce with what kit I have available)

@jmattsson
Copy link
Member Author

esp32c3 and esp32c6
port: usb-serial-jtag
behavior =>
uart.write(0) causes 'uart driver error' message
using
uart.start(0) -- no 'driver error' message, but
uart.write(0,'a\n') -- nothing is sent to the console

This is correct behaviour. If you select the USB-SERIAL-JTAG as the console, writing to UART0 will not go to the console. These are two separate hardware devices. USB-SERIAL-JTAG is not a UART. If you hook up an external UART-to-USB adapter to the UART0 pins you should see the "a\n" appear there.

esp32-s3 zero
Chip is ESP32-S3 (QFN56) (revision v0.1)
Features: WiFi, BLE, Embedded Flash 4MB (XMC), Embedded PSRAM 2MB (AP_3v3)

port: usb-serial-jtag
idf5.1.3 REPL -> O.K.
idf5.2.2 REPL -> O.K.
idf5.2.3 REPL, but typing any char in console reboots esp
idf5.3.1 repeated reboot

I don't currently have this dev board (I've ordered one now). Could you please post the crash log from make monitor on 5.3.1 here, and I can see if I can figure it out without hardware?

@serg3295
Copy link

Could you please post the crash log from make monitor on 5.3.1 here, and I can see if I can figure it out without hardware?

Crash log
esp32-S3 zero usb-serial-jtag
Serial port /dev/ttyACM0
Connecting...
Detecting chip type... ESP32-S3
Running idf_monitor in directory /home/serg/Projects/lua/nodeMCU-firmware
Executing "/home/serg/.espressif/python_env/idf5.3_py3.12_env/bin/python /home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/tools/idf_monitor.py -p /dev/ttyACM0 -b 115200 --toolchain-prefix riscv32-esp-elf- --target esp32c3 --revision 3 --decode-panic backtrace /home/serg/Projects/lua/nodeMCU-firmware/build/nodemcu.elf -m '/home/serg/.espressif/python_env/idf5.3_py3.12_env/bin/python' '/home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/tools/idf.py'"...
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40048dfc
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce2810,len:0x178c
load:0x403c8700,len:0x4
load:0x403c8704,len:0xc10
load:0x403cb700,len:0x2dc0
entry 0x403c8904
�[0;32mI (26) boot: ESP-IDF v5.3.1 2nd stage bootloader�[0m
�[0;32mI (27) boot: compile time Oct 17 2024 17:48:31�[0m
�[0;32mI (27) boot: Multicore bootloader�[0m
�[0;32mI (27) boot: chip revision: v0.1�[0m
�[0;32mI (27) boot.esp32s3: Boot SPI Speed : 80MHz�[0m
�[0;32mI (27) boot.esp32s3: SPI Mode       : DIO�[0m
�[0;32mI (28) boot.esp32s3: SPI Flash Size : 4MB�[0m
�[0;32mI (28) boot: Enabling RNG early entropy source...�[0m
�[0;32mI (28) boot: Partition Table:�[0m
�[0;32mI (28) boot: ## Label            Usage          Type ST Offset   Length�[0m
�[0;32mI (29) boot:  0 nvs              WiFi data        01 02 00009000 00006000�[0m
�[0;32mI (29) boot:  1 phy_init         RF data          01 01 0000f000 00001000�[0m
�[0;32mI (29) boot:  2 factory          factory app      00 00 00010000 00180000�[0m
�[0;32mI (30) boot:  3 lfs              unknown          c2 01 00190000 00010000�[0m
�[0;32mI (30) boot:  4 storage          Unknown data     01 82 001a0000 00070000�[0m
�[0;32mI (31) boot: End of partition table�[0m
�[0;32mI (31) esp_image: segment 0: paddr=00010020 vaddr=3c0c0020 size=2d5e8h (185832) map�[0m
�[0;32mI (65) esp_image: segment 1: paddr=0003d610 vaddr=3fc9ab00 size=02a08h ( 10760) load�[0m
�[0;32mI (67) esp_image: segment 2: paddr=00040020 vaddr=42000020 size=bf524h (783652) map�[0m
�[0;32mI (208) esp_image: segment 3: paddr=000ff54c vaddr=3fc9d508 size=01f5ch (  8028) load�[0m
�[0;32mI (210) esp_image: segment 4: paddr=001014b0 vaddr=40374000 size=16a20h ( 92704) load�[0m
�[0;32mI (231) esp_image: segment 5: paddr=00117ed8 vaddr=600fe000 size=00100h (   256) load�[0m
�[0;32mI (231) esp_image: segment 6: paddr=00117fe0 vaddr=600fe100 size=00008h (     8) load�[0m
�[0;32mI (241) boot: Loaded app from partition at offset 0x10000�[0m
�[0;32mI (241) boot: Disabling RNG early entropy source...�[0m
�[0;32mI (242) cpu_start: Multicore app�[0m
�[0;32mI (251) cpu_start: Pro cpu start user code�[0m
�[0;32mI (251) cpu_start: cpu freq: 160000000 Hz�[0m
�[0;32mI (251) app_init: Application information:�[0m
�[0;32mI (252) app_init: Project name:     nodemcu�[0m
�[0;32mI (252) app_init: App version:      tmr-libmain-binpatch150-891-gb7�[0m
�[0;32mI (252) app_init: Compile time:     Oct 17 2024 17:48:28�[0m
�[0;32mI (252) app_init: ELF file SHA256:  5cc304fc0...�[0m
�[0;32mI (252) app_init: ESP-IDF:          v5.3.1�[0m
�[0;32mI (253) efuse_init: Min chip rev:     v0.0�[0m
�[0;32mI (253) efuse_init: Max chip rev:     v0.99 �[0m
�[0;32mI (253) efuse_init: Chip rev:         v0.1�[0m
�[0;32mI (253) heap_init: Initializing. RAM available for dynamic allocation:�[0m
�[0;32mI (253) heap_init: At 3FCA3A48 len 00045CC8 (279 KiB): RAM�[0m
�[0;32mI (254) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM�[0m
�[0;32mI (254) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM�[0m
�[0;32mI (254) heap_init: At 600FE108 len 00001EE0 (7 KiB): RTCRAM�[0m
�[0;32mI (256) spi_flash: detected chip: generic�[0m
�[0;32mI (256) spi_flash: flash io: dio�[0m
�[0;33mW (256) rmt(legacy): legacy driver is deprecated, please migrate to `driver/rmt_tx.h` and/or `driver/rmt_rx.h`�[0m
�[0;33mW (256) i2c: This driver is an old driver, please migrate your application code to adapt `driver/i2c_master.h`�[0m
�[0;33mW (257) ADC: legacy driver is deprecated, please migrate to `esp_adc/adc_oneshot.h`�[0m
�[0;32mI (257) sleep: Configure to isolate all GPIO pins in sleep state�[0m
�[0;32mI (258) sleep: Enable automatic switching of GPIO sleep configuration�[0m
�[0;32mI (258) main_task: Started on CPU0�[0m
�[0;32mI (268) main_task: Calling app_main()�[0m

***ERROR*** A stack overflow in task console has been detected.


Backtrace: 0x40375c0e:0x3fcae220 0x4037e865:0x3fcae240 0x4037f572:0x3fcae260 0x40380aa7:0x3fcae2e0 0x4037f6a4:0x3fcae310 0x4037f69a:0x3fcae2fc |<-CORRUPTED



ELF file SHA256: 5cc304fc0

Rebooting...
ESP-ROM:esp32s3-20210327
debugger log
Set GDB target to 'esp32s3.cpu0'
2
[esp32s3.cpu1] Target halted, PC=0x4037BE8A, debug_reason=00000000
Set GDB target to 'esp32s3.cpu1'
[esp32s3.cpu0] Target halted, PC=0x40375C11, debug_reason=00000001
[esp32s3.cpu0] Halt cause (***ERROR*** A stack overflow in task console has been detected.)
[New Thread 1070261884]
[New Thread 1070259388]
[New Thread 1070251056]

Thread 9 "console" received signal SIGTRAP, Trace/breakpoint trap.
[Switching to Thread 1070261884]
0x40375c11 in panic_abort (details=0x3fcae1a0 "***ERROR*** A stack overflow in task console has been detected.") at /home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/components/esp_system/panic.c:463
463	    *((volatile int *) 0) = 0; // NOLINT(clang-analyzer-core.NullDereference) should be an invalid operation on targets

Solution

I have increased console_task stack to 1152 and the error has gone.

  xTaskCreate(
    console_task, "console", 1152, NULL, ESP_TASK_MAIN_PRIO+1, NULL);
}

@jmattsson
Copy link
Member Author

Thanks @serg3295 !

Alright, this is interesting. I was chasing down why I needed the explicit call to esp_vfs_console_register() when by all evidence it should not be needed because it's supposed to get automatically called from the low-level startup code. Yet without it I kept seeing crashes on ESP32 as soon as a character was read.

To my surprise, when I increased the stack size for the console task this issue too disappeared. Hmm. I'm not sure whether this makes me happy or deeply unhappy... I didn't have either of the stack overflow check methods complain for me.

This allegedly resolves the reboot crashes seen on ESP32-S3 Zero.
It also removes the peculiar need to explicitly call
esp_vfs_console_register(), the absence of which previously would
trigger an assertion failure in xQueueGenericSend on input.
@serg3295
Copy link

serg3295 commented Oct 18, 2024

If you hook up an external UART-to-USB adapter to the UART0 pins you should see the "a\n" appear there.

I see. Thanks. I received a message to the UART0 connected via the FTDI adapter

If I understand correctly, the only way to send a message to the CDC and JTAG console from esp is 'print()'?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants