A Bunch of Questions from First Time MODM User

[thestumbler]

Okay, I've been experimenting with MODM for a few days, and been really impressed so far. But I've accumulated a bunch of questions, and haven't found the answers in brief searches (it may be I'm searching in the wrong places). I should also point out that while I'm comfortable with C and Python, I've only been applying C++ to embedded systems for the past year or so, and am completely unfamiliar with the LBUILD system. I suspect a bunch of my questions can be solved by digging deeper into that. If this laundry list of questions would be better to post on issues, and/or broken up into multiple questions, let me know.

Installation

The directions in the README of the main repo correctly call out a recursive git clone of the MODM repository. But the instructions for making a custom project only call out the basic git submodule add command. I think you need to also do a git submodule update --init --recursive after the initial add.

Custom BSP

So far I'm experimenting with standard dev boards, but I will need to define a custom BSP. I follow the gist of the instructions for making a BSP, but one thing confuses me. The instructions offer two approaches, copy and modify an existing BSP, or make your own completely from scratch. I'm shooting for the former. The instructions hint at copying an existing BDP directory into your own project and modifying it. I can't figure out how to make that work. What was successful was to make a new BSP directory in the MODM tree, but that causes another problem (see FORKING below).

I may be misreading the custom BSP instructions, because it also could be interpreted as "copy all the stuff from an existing BSP and insert it (unclear how/where) in your own project. I started with that approach, actually, and very quickly realized I didn't know enough. That's when I tried just duplicating an existing BSP in the native MODM directory tree, which seems to work.

How are you supposed to use a BSP directory that is copied / modified into your project's directory tree structure?

Inherit

A couple places the docs talk about "inheriting" the results of lbuild discover into your project.xml file.
At first I thought this just meant manual copy/pasting, but after rereading it a few times, I wonder if there is some specific INHERIT operation I'm not understanding that does that automatically.

Project Directory Tree Structure

I was a little unclear about some points of the recommended tree structure. I thought that I would browse some of the existing MODM projects listed here. I couldn't find any project that used the recommended structure. Anyway, I think I have a good grasp on that now.

Compiler / Linker Tweaks

I found the ability to add custom CFLAGS and CCFLAGS into the project.xml file. But I need to do more. In particular, I am porting a small boot loader to use MODM, and it requires a few special things. One of those is a few fixed-address variables. I'm porting this from ARM's embedded compiler (Keil), where we had things like

const char bootloader_version[BLVERSION_SIZE] __attribute__((at(0x8000200))) = VERSION;

I figured out how to do this in ARM-NONE-EABI-GCC/G++, but it will require tweaking the linker load script. I'm hoping I can do that from the .xml file rather than tweaking the SCONS or MAKE files post LBUILD.

UART Queues

One common thing is to know how many bytes are in the UART queues. It doesn't look like the normal UART class exposes this to the application. I haven't tried yet, but the Queue class does have a getSize() function which seems to be what I'm looking for. Can I just reach over the UART class and grab it?
Board::stlink::Uart::rxBuffer::getSize()??? Also, what's the significance of the stlink namespace for the UART? In my custom board, I also use the same PD8/9 pins, although there may or may not be an STLINK attached to them. It that name arbitrary? Or does deeper stuff in MODM depend on it?

Forking

So, because of the recommended directory tree, MODM is now in your project as a submodule. That's fine. But when I thought I had to modify the boards directory, it opens a can of worms. I forked MODM so that I could push my own modified copy. But as I quickly discovered, MODM itself is comprised of a bunch of submodules. In the case of modifying the board folder, that's in the main MODM repo. But if one needed to tweak any of the submodules of MODM, that gets messy -- you have to fork each submodule of interest as well.

What's the intended approach to forking the MODM repo, with respect to the submodules? Hopefully, I won't care about this and find out I can put my modified BSP in my project folder instead.

Overall, I'm finding the documentation adequate. The actual API stuff you need in your application is documented the best. It's the documentation on the surrounding bits and pieces that I'm missing. And most of that may be my unfamiliarity with LBUILD. The Guide section is really good and helpful, but I think it could be expanded a little. That's all for now. Next up is to figure out how to talk to the NVIC peripheral.

EDIT:

I forgot one thing. In my makefiles, I usually have more granular control that seems possible from the LBUILD-generated makefile. For example, I can't just compile one file, like make packet.o. Is there a way to enable this from LBUILD?

Also, when it comes to programming my chip from the makefile, I usually add a bunch of targets. For example:

Flash operations:
=================
info     gives summary info on target
write    uploades .bin file into the target
erase    does a mass erase
reset    resets the target
trace    starts trace for SWO debug printf
-----------------
usart-info      gives summary info on target
usart-write     uploades .hex file into the target
usart-erase     does a mass erase
usart-reset     resets the target

I can only see the PROGRAM option being generated from LBUILD. Is there a way to add my own targets, like above?

[salkinium]

I think you need to also do a git submodule update --init --recursive after the initial add.

Yes, good point, do you want to add this directly to the documentation as your first PR? You don't have to, it's just a neat way to contribute and you can do it entirely online.

How are you supposed to use a BSP directory that is copied / modified into your project's directory tree structure?

BSPs are not special in modm, they are a configuration file that you can extend and a bunch of files that get included automatically.
The file are the easy part, that's just a copy to your application folder, but the configuration you must now move into your project.xml. Have a look at what modules, options the board.xml and what dependencies the module.lb has for your closest BSP, all that configuration needs to move to the project.xml. (example: DISCO-F469)

But I recommend to just use your fork with the customized BSP inside modm, it's the path of least resistance, particularly when your new. It's also not wrong to do it that way.

I wonder if there is some specific INHERIT operation I'm not understanding that does that automatically.

Ah, that's indeed a bit unclear. This refers to the <extend>alias-or-path/to/project.xml</extends> tags inside project.xml, which allows you to inherit the configuration from another file. This is how BSPs are implemented, the <extends>modm:disco-f469ni:b-03</extends> alias expands to <extends>modm/src/modm/board/disco_f469ni/board.xml</extends>. You can inspect this by doing lbuild -v discover, which will print:

 $ lbuild -v discover
[INFO] lbuild.parser:
ConfigNode(filename='project.xml')
╰── ConfigNode(filename='../../../src/modm/board/disco_f469ni/board.xml')
    ╰── ConfigNode(filename='../../lbuild.xml')

The code that builds the alias list is here btw. It also autodiscovers your custom BSP inside modm.

I couldn't find any project that used the recommended structure.

Ooops. I think we added this guideline relatively late and a bunch of people already started doing their own organization. We should really resurrect our git template that you could just fork to get started. GitHub had an issue for several years where submodules in templates did not get forked correctly, but I think they fixed it now.

I'm hoping I can do that from the .xml file

Yes, you can overwrite the linkerscript path with the modm:build:linkerscript.override option, then provide your completely own custom linkerscript (ideally by modifying the generated).
For your non-bootloader application, you also need to use the modm:build:linkerscript.flash_offset option to move everything beyond the bootloader.

Sadly GNU ld is not as powerful as the commercial embedded linkers from Keil or Segger. Flood filling, compression, explicit placement policies are all not really available… sad.

Can I just reach over the UART class and grab it?

Nope, the interface was not build for that. They are not even publicly linked. I would recommend copying the generated UART driver into your application and modifying it. modm is very modular, so you can replace the modm::platform::UartN with your own, since it's just a template. Make sure to disable the driver in lbuild again, otherwise it'll hog the interrupt handlers. Alternatively modify your fork, and open a PR, cos I think it would be a useful addition to be able to query the queue state.

Also, what's the significance of the stlink namespace for the UART?

It's just a way to point out that the uart is used exclusively for the STLink that's soldered onto the dev board. You don't need to follow that naming scheme for your own board, nothing in modm depends on that. The board uses it to define the logger and connect that to the IO streams. You have to define that manually for your board.

But if one needed to tweak any of the submodules of MODM, that gets messy -- you have to fork each submodule of interest as well.

True, but all our submodules are external projects that are pruned to minimal size. We usually try to avoid forking external projects. If we need to modify something, we can use lbuild to add/replace files. All external submodule projects are automatically updated in modm-ext.

What's the intended approach to forking the MODM repo, with respect to the submodules?

You shouldn't need to touch them. Your modm fork will just use the submodules, they don't need to be forked themselves. If you want to add an external project, here's a small guide.

And most of that may be my unfamiliarity with LBUILD.

lbuild has a bunch of documentation itself. It's basically a modern take on the typical KConfig + CPP combo: here it's Python + Jinja2. It just allows way more flexibility in managing such a complex configuration for thousands of targets. Also way easier to generate docs from it ;-P

Next up is to figure out how to talk to the NVIC peripheral.

(modm implements the CMSIS functions.)

In my makefiles, I usually have more granular control that seems possible from the LBUILD-generated makefile.

We typically use SCons, but it's quite inscrutible sometimes. Therefore I added the modm:build:make module that generates a simple Makefile only build system.
I think you may need to prefix your file with the build path, so maybe make build/packet.o will work? The application sources are globbed here and then transformed to .o files here. There's probably a more elegant way to doing that, but it works for now.

Is there a way to add my own targets, like above?

Sure, you can modify the Makefile and set the modm:build:make:include_makefile option to false to prevent lbuild from overwriting it. It's BSD licensed specifically for the purpose of modifiction.
However, note that the flash operations are all done by calling the Python library modm_tools. That way we can use the same tools for SCons, Make, and CMake. Their APIs are documented here.

[thestumbler]

Makefiles: Yeah, I spoke to soon about them. I did figure out you had already included helpful target like erase and reset. In scons as well.

Instead of telling LBUILD not to overwrite the Makefile, which can be useful thing sometimes, is there a more native LBUILD-like way of doing this? Could I put a store a file of my custom Makefile targets and tell LBUILD to add that to the Makefile (ditto, SCONS)? That's essentially what I'm doing manually anyway, I just copy and paste from another project.

Funny comment I got from a colleague yesterday. I asked her if she had any experience or heard of folks using SCONS in Korea.
She said she had never heard of that except in a bakery. It took me a minute to figure out what she was talking about -- she misread SCONS as SCONES

[salkinium]

No, lbuild already does too much and build system customization is too complicated to abstract. The escape hatch is adapting the top-level Makefile/SConstruct and telling lbuild not to overwrite it. We recommend that you commit the generated sources incl. the Makefile for that purpose (and to check what changed when upgrading modm).

[thestumbler]

Can I just reach over the UART class and grab it?

Nope, the interface was not build for that. They are not even publicly linked. I would recommend copying the generated UART driver into your application and modifying it. modm is very modular, so you can replace the modm::platform::UartN with your own, since it's just a template. Make sure to disable the driver in lbuild again, otherwise it'll hog the interrupt handlers. Alternatively modify your fork, and open a PR, cos I think it would be a useful addition to be able to query the queue state.

Initially I postponed dealing with this issue. Last night I couldn't delay any longer because I needed to grab the disableOperations() function as part of the sequence of critical steps you do when you bring everything to a grinding halt and jump to a new start address with a clean slate. This can be used to jump to an app or to jump back to the boot loader from an app.

After some trial and error, I stumbled upon this approach that seemed to be the path of least resistance. In the board.hpp file making that STLINK namespace, I added one more entry

UartHal = Uart3_Hal

And then I can do things like disable the UART from my app using Board::stlink::UartHal::disableOperation()