Freeze of 4.3 and what's next

[philippe44]

We have agreed that it was time to freeze master-v4.3 and move on to a new base. So we'll make a few changes in 4.3 to roll back some recent updates and move them to a new branch and then we'll start fresh from there. The master-v4.3 will be bug correction only. One of the main motivation is the limitation of memory partitioning in 4.3 which is becoming too painful for us - and fun has to be part of the deal 😄. So next version will require a full download of the memory, there will be no OTA upgrade.

A few things we have in mind

• move to esp-idf 4.4 so that we have full compatibility between esp32 and esp32-s3
• new memory partition so 8MB of flash will be the minimum required size for esp32's (tbc)
• refactor the config management so that the UI can expose all features consistently
• gather statistics to help us better understanding what's being used and how (respecting privacy of course)
• update codec management to add ADC and line-in
• include microPython so that you can add your own code (as long as hooks to the existing code exist 😄) ?
• multiple AP ?
• https web access ?

Comments welcome

[ElFishi]

Is there a way to check the memory size of a current system either by CLI or webUI?

[sle118]

The webui also has a way to view this.
1- activate the advanced option under credits
2- navigate to the advanced tab
3- Execute "Get version of chip and SDK"
https://photos.app.goo.gl/XD8cYhQPU6oHYGHJ6

[philippe44]

At boot, it should be amongst the first few lines

[ElFishi]

W (1852) nvs_utilities: Configuration memory usage. Heap internal:227071 (min:226931) (used:216) external:4054483 (min:4046551) (used:16272)
Am I right that this means I can't upgrade to 4.4 in the present configuration?

[philippe44]

I (29) boot: ESP-IDF v4.3.5-144-gbcec78f672 2nd stage bootloader
I (29) boot: compile time 10:45:32
I (29) boot: chip revision: v1.0
I (34) qio_mode: Enabling default flash chip QIO
I (39) boot.esp32: SPI Speed : 80MHz
I (44) boot.esp32: SPI Mode : QIO
I (48) boot.esp32: SPI Flash Size : 4MB
I (53) boot: Enabling RNG early entropy source...

[ElFishi]

Oh, so I need to hook up a serial interface.

[ElFishi]

ets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:6948
load:0x40078000,len:13772
load:0x40080400,len:4532
entry 0x400806a0
I (62) boot: Chip Revision: 1
I (63) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (39) boot: ESP-IDF v4.0.2-206-g5630b17e1 2nd stage bootloader
I (39) boot: compile time 05:46:08
I (39) boot: Enabling RNG early entropy source...
I (45) qio_mode: Enabling default flash chip QIO
I (50) boot: SPI Speed      : 80MHz
I (54) boot: SPI Mode       : QIO
I (58) boot: SPI Flash Size : 4MB
I (62) boot: Partition Table:
I (66) boot: ## Label            Usage          Type ST Offset   Length
I (73) boot:  0 nvs              WiFi data        01 02 00009000 00004000
I (81) boot:  1 otadata          OTA data         01 00 0000d000 00002000
I (88) boot:  2 phy_init         RF data          01 01 0000f000 00001000
I (96) boot:  3 recovery         factory app      00 00 00010000 00140000
I (103) boot:  4 ota_0            OTA app          00 10 00150000 002a0000
I (111) boot:  5 settings         WiFi data        01 02 003f0000 00010000
I (118) boot: End of partition table
I (123) boot_comm: chip revision: 1, min. application chip revision: 0

too small, right?

[ElFishi]

Can I please ask, the memory requirement of 8MB pertains to both flash and PSRAM? Or only PSRAM?
I try to identify which ESP32-S3 modules I shall get. Looking at the WROOM I see

Ordering Code           Flash            PSRAM
ESP32-S3-WROOM-1-N4      4 MB (Quad SPI)  - 
ESP32-S3-WROOM-1-N8      8 MB (Quad SPI)  - 
ESP32-S3-WROOM-1-N16    16 MB (Quad SPI)  - 
ESP32-S3-WROOM-1-H4      4 MB (Quad SPI)  -
ESP32-S3-WROOM-1-N4R2    4 MB (Quad SPI)  2 MB (Quad SPI) 
ESP32-S3-WROOM-1-N8R2    8 MB (Quad SPI)  2 MB (Quad SPI) 
ESP32-S3-WROOM-1-N16R2  16 MB (Quad SPI)  2 MB (Quad SPI) 
ESP32-S3-WROOM-1-N4R8    4 MB (Quad SPI)  8 MB (Octal SPI) 
ESP32-S3-WROOM-1-N8R8    8 MB (Quad SPI)  8 MB (Octal SPI) 
ESP32-S3-WROOM-1-N16R8  16 MB (Quad SPI)  8 MB (Octal SPI)  

Will N4 do, or does it have to be N8 or N16?
Or does it have to be R8, or both?

[sle118]

The requirement is mainly focused on flash for now, as it is the main limitation for code space. For psram, octal spi is better if you want to play with the upcoming micro Python, as it significantly increase performance.

[philippe44]

I think there is an octal Flash as well, no?

[ElFishi]

not in the WROOM datasheet

[philippe44]

That's the one you want https://www.espressif.com/sites/default/files/documentation/esp32-s3-wroom-2_datasheet_en.pdf

[ElFishi]

Oh, OK.

[sle118]

• include microPython so that you can add your own code (as long as hooks to the existing code exist 😄) ?

This opens the door to interesting feature deployment mechanisms which doesn't require a full reflash of the code. For example, supporting different types of i2c DAC beyond what the current control set allows, hooking up to events in the system.

The web UI is another example. Instead of baking it in one of the binary partition, I will move it to a new file partition that is shared between all executable partitions. This will open the door to real OTA with no recovery. You upgrade the "other partition" and boot it up. Next release updates the previous one and so on. So technically, it will be possible to switch between two versions of squeezelite residing on flash.

As for configuration, I think that we should preserve the current NVS approach as I've built a universal parser/tokenizer that allows keeping it this way, but if you are opening the door to something else, I think a full JSON or yaml could be better... Although YAML is temperamental with alignments

Moving to the esp-idf v4.4 also means test automation can be compiled for a mock SUT under linux, opening the door to improved stability and quick debugging of new features (no jtag needed). For this to work, I'll need some help with writing test cases for older components.
https://docs.espressif.com/projects/esp-idf/en/v4.4.6/esp32/api-guides/unit-tests.html

[sle118]

For the record, am currently experimenting with protocol buffers, as it is already being used in our system and thus we could benefit from already linked pieces of the framework. It is not as human readable as JSON, but definitely more efficient and it also supports Json serialization, which I would do on the front end.

[DerPicknicker]

Why not using IDF 5...? It's already compatible with ADF. I used that combination together.

[philippe44]

It's a massive change on many API, the whole networking stack AFAIK is to be redone. So that's not a very exciting promise

[DerPicknicker]

Oh. I can understand that. But at some
Point you should think about. IDF 4 is at some point EOL with no future updates. Just have this in Mind.

[philippe44]

yes sir 😄 but it's just @sle118 and @philippe44, the two of us (with increasing help from @wizmo2)

[DerPicknicker]

@philippe44

Full ACK that you want to focus on things which are related to the use of the users. Under the hood changes are Cool from the technical perspective but somehow useless if the user won't see them 😅👋🏻

[wizmo2]

Re. Line-In progress
I have 'absolutely' hacked in functioning mic/line-in capabilities!

I have it embedded as another sink "app" using the external-decoder interface. When enabled, the app starts on boot and continuously streaming raw WAV/PCM format over UDP to a remote host. I enabled the Boom Line-in functionality in the SqueezeESP32 plugin so that it can be control with LMS. This includes loopback functionality to feedback audio to the DAC.

Up 'till now, I've been concentrating on the microphone input and making the stream compatible with Rhasspy and openWakeWord. My head hurts, but it does now work and is stable (but a long way from being shareable), The HAL input is stereo, so Line-In should work with simple mixer changes.

Next steps are

• tidy up code.
• stream encoders (?libflac?) that could potentially be LMS compatible.
• I'm going to investigate the ADAC configuration some more, but will need help on this. IMO, I think we may need to look at moving this to a shareable component. (NOTE: My T-Embed S3 dev-kit has a separate ES7210 I2S chip for the microphone, so I've been able to use the 2nd I2S channel to keep ADC fully independent from the DAC. To make it applicable to other devices such as the A1S AC101, it will take a fair amount of thought, if it's even possible)

This is mostly just a FYI. Still a lot to learn, but am at least making some progress, Two things I have discovered it that a) ESP-IDF is extremely frustrating without JTAG (next task!), and b) every road-block you do hit, @philippe44 has already been there!

[philippe44]

That's great progress. One thing I have to warn you about, JTAG is very frustrating on esp32. Most of the time, unless there is no other way, I prefer debug by logging

[sle118]

compiler optimization will pretty much end up inlining some code and make variables disappear under "optimized out". That is when you are actually able to reach debugging. Some years ago I added debugging script building to the make system, so look inside of the build folder for them. They contain the necessary instructions to properly tell the debugger about the offsets for debugging the ota_0 partition (squeezelite).

[wizmo2]

Had not really looked into the limitations. Was hoping it was going to solve a lot of issues, but it sounds like it may just be more of a headache! Thanks for the heads up!

[philippe44]

It can help, for sure but it's far from being as slick as a debugger on a regular host

[philippe44]

We are discussing with @sle118 an option to keep 4MB possible, but it will require a *full redownload, so a serial cable will be needed (if we find a solution)

[DerPicknicker]

Hi @philippe44 ..

What does this mean in general? Which esp32 aren't supported anymore? Does it only mean that I have a model with a bigger PSRAM or also with a bigger internal flash. If the flash must be bigger I see a lot of problems. Because if you order models on AliExpress or other sources you can't trust the size of the flash.

Also for the hardware which is already in use. I hope the best for using only 4mb of flash. Or as an alternative make it possible to use different binary's which doesn't support all of the features. For example I don't need displays why should my binary include the display code? This would help to make it possible to reduce the binary size and makes it possible to support also smaller 4mb models.

Edit: this approach is already used by several other projects for example Tasmota.

[philippe44]

We'll see but this might mean that the 4MB modules will be stuck with just the bug fixes of the current version. After a couple of discussion, we might be able to avoid doing that, but that will require a lot more complicated refactoring and re-architecture so we need to put more thoughts into that.

[DerPicknicker]

@philippe44

Thank you for the explanation. For me it's unclear why you're not offering different binary's which are much smaller than one big one..? Is that not possible due the software architecture or much more effort on your side?

[philippe44]

I'm not saying we will not but understand that all this is work and time and maintaining multiple version can be exponentially difficult

[sle118]

As for the 4mb limitation, it's not set in stone yet. It will depend on how much room I manage to squeeze (pun intended) with the configuration refactoring. After all I still own nice 4mb devices here. The main reason for aiming at a larger flash is also to give us some room for interesting features. For example, @philippe44 suggested the implementation of a script interpreter, which would open the door to a lot of possibilities like DAC init done in scripts, controlling gpio's from system events, etc. Which looks really fun to play with.

[DerPicknicker]

Hi,

thank you for the great piece of software.
Here are my thoughts:

• Strip Down the WebUI and hide all unnecessary features for beginners
• This meant that you should have an easy mode there you only select WiFi, select the presets of the used hardware
• If there are settings which are not to change for a normal user hide them and only show them if you enabled the "Advanced View" .. Because I never change the default settings and I use the Ui only to change the WiFi, update or reset it to defaults
• Make the mapping of new (unsupported) hardware easier as well as the Mapping for Display,Button, Rotary-Encoders GPIO eg. => A Dropdown in the UI, for each hardware Component

This would my ideas at the moment for making this software even better. HTTPS is completely useless for me. All my Webuis are http and accessed by the IP-Adress. Using https in a local network makes no sense for me. If I want to access the webui from the outside I use an WireGuard VPN. I would focus more on making things idiot-proof 😅

Last But not least:
I would think about to make donation possible. I wrote this in a different post already. This software makes things so much easier and helps to build an real alternative to Sonos and other Multiroom systems. A small donation is only fair.

For me also it would be a interesting idea: All who donated to the project can vote for future features which you suggested. To give them a little value back (in terms of giving them a voice)

Thank you for reading this long post.

[DerPicknicker]

@sle118 ...
Thank you for the detailed explanation of your next steps. I can only say things from the user perspective. An ease of use is for me more important than the best software with a lot of features and functions which I only can use with reading a lot of documents and other stuff.

And I can imagine how hard it is to get a this stuff running on a simple esp32.

[philippe44]

On the topic of donation, I appreciate the idea but here is my very personal view on this:

• Either you can make a good living out that and yes, this is worth doing but then it's not a hobby anymore, it's a job like any others and that has many implications.
• Or you cannot make a living out of that (which is the most likely case) and you are stuck between a rock and a hard place; you have created a system where people feel entitled to support and get answers and get their wishes fulfilled but they don't realize that the money they are providing is covering a small fraction of the cost of the work.

Free means no ties on both ends and that's the way I like it. I even extended that to my SqueezeAMP project where people only pay for what it costs me, from a pure bill of material point of view. The contract is clear. You're not paying me, you're paying for the parts and the cost others are charging me for doing the assembly, shipment, taxes and so on. I'm happy to support you with best intent, but that's all. You did not buy any right of any kind. If you don't like the contract, that's fine, don't get one.

Again, it's my own PoV, not the reflection of a discussion with @sle118.

[sle118]

I could not agree more to this view. Not that I wouldn't want to live from this, but that would be unsustainable given the market size.

[DerPicknicker]

Thank you @philippe44 for the detailed explanation of your POV. And also @sle118 for responding to this topic.

I can totally understand your Thoughts and Opinions about donation. For me it's a crucial topic to show support. What I've learned is that you want support in a different way. I want to support you, if I can test something out. You can always ask me and what I also do is writing some tutorials about LMS and your software to make it more public in the community. I can also help with translations from English to German (if not already done)

** Last but not least: **
I came up with another idea to make it more idiotic-Proof 😅... The setup of the device could be guided. Like setup a new phone: 1. Setup WiFi, 2. Select the Hostname, 3. Select the Hardware, 4. Activate additional features (Display, Buttons eG..) 5. You're done.

[philippe44]

And I really do appreciate the intent of the donation, thank you. Maybe one day @sle118 and I we'll make all this a business venture 😂.

Having said that, for me, the most valuable thing you can bring is help for test & validation. This is always the part that developers grossly underestimate but this is also very time consuming when you need to try all permutation and combination of everything. And BTW time is the most precious resource of life (IMHO). Also, with tests done by others, it brings a perspective that I would never have had and that helps finding issue much more efficiently.

After test, come all sort of user guides, how-to and application notes. These are also hyper-important. You might see that this project has a lengthy README because I believe this is critical for success. Yes, nobody RTFM, except when the shit hits the fan and then it's a life saver. So contribution to improve README (or do other guides) would be really welcome.

[sle118]

If anyone is interested, here's a link to the current state of the new configuration object.

Updated as of 2023/11/28

[sle118]

and the system will communicate its status via this structure (work in progress)

[wizmo2]

Re. Line-In progress
Most of the clean up is done. The ADC sink app with low quality microphone stream works well and uses minimal resources. It's a nice feature to be able to use your existing player as a VA room satellite, I've ordered a couple of INMP441 modules to test solutions for generic boards.

I do have a working solution for A1S(AC101) and 'Generic I2S' sharing the existing DAC, but there are numerous issues. Some are resolvable, but I coming to the realization that trying to stream and encode/decode 2 HQ streams on the same device does not make sense, and Line-In streaming is probably better suited to a dedicated device with hardware to match.

I'm going to concentrate on the dedicated ADC I2S solutions, for now

Am interested on getting feedback on the interest level for

• local line-in playback only (output mixer controls) for existing DAC I2S.
• an alternate build for a ESP32 based line-in streamer compatible with LMS.

[sle118]

Your conclusions explain the hardware choices I made for the SqueezeIO, which has a DAC chip with an internal mixer. This allows pushing analog, optical and coax audio coming from a line in (I could even have spdif in) while encoding the incoming i2s stream for streaming. I have not tried line in yet because I haven't had the opportunity to assemble the 2nd version of my board yet (December, here I come).

Local mixing poses another challenge, though, as local latency will be much lower than other players. This probably means you won't be able to sync nicely with a player in the same room, but it is probably going to be acceptable for other rooms given the distance. The fact that the SqueezeIO also has ethernet will help, as this tends to reduce latency a bit.

[formatBCE]

@wizmo2 hi! Your post gave me hope. I see you're about to include mics to the build. Probably, you have some code already to make HA voice assistant possible?
Wyoming protocol seems to be clear, and there is ESPHome implementation for voice assistant (I managed also make simple media player on same device, using custom i2s component), but this project would replace existing Alexa/Google devices completely. I have almost no experience in low-level embedded programming, but I really, really want something like this, and ready to help with it.

[wizmo2]

Probably, you have some code already to make HA voice assistant possible?

The code in my fork supports the original version of Rhasspy and OpenWakeWord (pre Wyoming) which I am actively using in in HA with the TTGO T-Embed players, There are other build versions available from my web installer, but no guarantees!

I had intended to move to Wyoming, but the project is currently on hold until
a. @sle118 is ready to share the new restructure branch, and/or
b. I have some time to look at it.

[formatBCE]

Yeah, Rhasspy was great, but we have new standard. :))

I will wait (and maybe dig into it myself a bit, although I'm not embedded dev in any way). Let me know, if or when you need any help with testing or brainstorming.

[wizmo2]

Do you guys have a docker image or docker file for 4.4?

[sle118]

I didn't create one yet. The "replatformed" branch will have different requirements for the build process with the introduction of protocol buffers and some python utilities that I have created to support my goals.

[DerPicknicker]

May I ask you about your background..? It seems crazy what stuff you do here:

• Docker build setup
• Multiroom audio on an esp32
• Webgui
• Using the ADF / IDF with the lack of documentation sometimes ....

What I am saying it seems crazy that you're only 2 guys. How many years you're already in software development especially with an esp32?

[sle118]

I've been a ERP freelance developer for many years, working with C mostly, but also a good number or programming languages on a side variety of development environments before shifting to the SAP ERP some years ago. As for the esp-idf, I had seen bits of it when playing with the Arduino platform, but decided to learn it for this project when we started it. The fact that I'm slowly doing a full refactoring is a bit because of this learning along the way; back when the project was being rooted on the ESP32, I had a vision of what I wanted but didn't have enough exposure to all the difference technologies I needed to achieve it. Webpack, node.js, protocol buffers, etc. So now I have a better understanding of our new and also more experience on these tools and the next platform will be more efficient and will have a better cohesion.

I typically learn by reading code to supplement the documentation, and I'm not shamed to admit that a chatgpt 4 subscription is an amazing tool for exploring new technologies and getting pointers to start in the right direction.

And most of @philippe44 code will stay as I like his style and his contributions are always really clean

[DerPicknicker]

Hello together,

after using the ESPLouder for some time. I thought about a new feature. I don't know how much cpu time you're already using.

This device has 2 speaker outputs. For me it would be awesome if the 2 channels would appear as different players in LMS. This would allow to create a multi speaker setup at a much lower cost.

Do you think this is possible if you don't use other components like displays, LED stripes and so on? This would be absolutely amazing if this is possible.

[sle118]

This device has 2 speaker outputs. For me it would be awesome if the 2 channels would appear as different players in LMS. This would allow to create a multi speaker setup at a much lower cost.

Very, very unlikely to happen. The underlying architecture wasn't built with this in mind and the hardware would probably not support that either; you would be better served with an old raspberry pi (or a clone) with multiple usb audio cards connected to it.

[DerPicknicker]

@sle118 ...

Cool to know. I will think about that. It was just an idea which came into my mind.

[bungh0l10]

Hi y'all, I'm just curious about your new project. I'm afraid to ask but can you already share some news? 🙂

[sle118]

I spent a lot of time during the December break on this and I am getting close to being satisfied enough with the code state to push the changes to a new branch. This is a very early stage, though. Some limitations at this point:

• No UI
• No OTA. In fact, I am still not certain on how I will deal with the upgrade process now, with the app partition like we have today, but with a UI that now sits in a file system and that is shared between Recovery and the application itself.
• Some system pieces need to be realigned with the new config structure.

I'd say that 90% of the functionality is there in the backend. When the code is pushed, there will be no support for it until we reach a state where we release it to the general "public". Developers will be welcomed to try and possibly contribute, but the code is far from being usable as a "daily driver".

[LeoSum8]

Hi all,
I am excited to see that you are working on a restructure. Especially the increased ease of creating "plugins" will be nice.

@philippe44 mentioned here, that disabling LMS is really not foreseen for this project, as it is LMS-centered and that

for example the audio backend is inside squeezelite so Airplay, BT and Spotify are nasty hacks I've added on top of it.

Is this something that will change with the restructuring?
I fully understand that this is still LMS-centered and don't ask you to change this. But as a non-LMS user that wants to use an efficient esp32 for music-streaming, this project currently seems the most active with the most dedicated devs :)

[sle118]

The restructuring is centered around the system's data model which leveraged a primitive data storage that led to parsing code duplication and a some overhead every time we needed to access the configuration/persistent state data. The new design relies on a centralized parsing and portable data structure definition that is based on Protocol Buffers and should make it easier to maintain consistency across all the components.

There are many reasons why the audio backend is implemented inside of squeezelite and it would be unlikely to change in the near future because this would require an even deeper refactoring effort, and I am not even sure if the current breed of the ESP32-Wrover would have enough processing power to support that. @philippe44 could probably give a more logical explanation here but, in short, we have to use multiple buffers to handle the audio data (streaming from the network, decoding audio, sending audio to the target device (i2c, etc.) and this takes up some significant bandwidth already. Moving away from a squeezelite centered system means we would likely have to create a lower level audio stream (another buffer) and transfer its control to a new component of the system. This means Squeezelite would be a simple user of the audio buffer rather than being at the center of it. To some extent, this is similar to how squeezelite works on Linux (ALSA) or Windows, where you can mix various signals and levels in and the master volume out. This is just the tip of the iceberg, though.

From a pure performance standpoint, there are some ESP32 which support octal PSRAM and some early tests show that it significantly increases the memory bandwidth to a point where the processor usage drops considerably. This is good news for whoever would like to design a system from scratch, but we are trying to keep backwards compatibility as much as possible here not only for the benefit of our users, but also for ourselves (myself), having several ESP32-Wrover devices spread across my home.

Last but not the least, we could certainly make it "appear" as though you aren't using Squeezelite in the background by keeping the streaming core alive while suppressing some unwanted side effects of not having a LMS active on a given network. @philippe44 already made some changes that allow this, but I just could'nt find the patience to expose it in a easy way on the UI because we are so tight in space on the current build; I tried but lost patience chasing little chunks of bytes here and there. This is part of the reason why I had no more fun with the project and part of the reason why @philippe44 and myself decided to freeze the current branch so I could start the refactoring.