Add flexible file system and POSIX file API support

[oyama]

This pull request adds flexible file system and POSIX file API support to pico-sdk. This PR fixes the following Issue:
#531
#767
#1663

The file system consists of several libraries, starting with the pico_filesystem library. Users are free to build the file system they need for their embedded projects using the familiar POSIX and C standard file API.

The pico_filesystem library contains two block devices with on-board flash memory and SPI connection SD card support. The file systems available are FAT and littlefs. The pico_filesystem library is enabled by simply appending the following function to CMakeLists.txt in the user project:

pico_enable_filesystem(${CMAKE_PROJECT_NAME})

Users can programmatically #include <pico/filesystem.h> and call fs_init() to manipulate files with the familiar POSIX file API:

#include <stdio.h>
#include <pico/stdlib.h>
#include <pico/filesystem.h>

int main(void) {
    stdio_init_all();
    fs_init();

    FILE *fp = fopen("/HELLO.TXT", "w");
    fprintf(fp, "Hello World!\n");
    fclose(fp);
}

In the default configuration, littlefs with onboard flash memory as block device is mounted in /. I have prepared a sample project pico-filesystem-examples repository that uses this actual diff, see also here.

Libraries to be added

The pico_filesystem consists of the block device abstraction library, the file system abstraction library and the virtual file system library pico_filesystem, which integrates these as Newlib system calls.

• pico_filesystem: POSIX and C standard file API, and File system management API libraries
• pico_filesystem_deafult: Convenience library retaining default file system configuration
• pico_filesystem_blockdevice_flash: On-board flash memory block device library
• pico_filesystem_blockdevice_sd: SPI-connected SD card block device library
• pico_filesystem_blockdevice_heap: Heap memory block device library
• pico_filesystem_filesystem_fat: FAT file system library (using FatFs)
• pico_filesystem_filesystem_littlefs: Littlefs file system library (using littlefs)

Any block device can be combined with any file system. Users can select only the libraries they need and add them to their projects.

Configuration filesystem

Block devices and file systems are each abstracted and can be freely combined. For example, the following combinations are possible:.

• / for littlefs on flash and /sd for FAT on SD cards.
• Format and use the flash with 512 KB littlefs and 512 KB FAT.
• Use SD card formatted with littlefs

To set a non-default file system, write CMakeLists.txt as follows:.

pico_enable_filesystem(${CMAKE_PROJECT_NAME} FS_INIT my_fs_init.c)

You can freely design your own file system with your own fs_init() function.

#include <stdio.h>
#include <string.h>
#include <hardware/clocks.h>
#include <hardware/flash.h>
#include <pico/filesystem.h>
#include <pico/filesystem/blockdevice/flash.h>
#include <pico/filesystem/blockdevice/sd.h>
#include <pico/filesystem/filesystem/fat.h>
#include <pico/filesystem/filesystem/littlefs.h>

bool fs_init(void) {
    blockdevice_t *flash = blockdevice_flash_create(PICO_FLASH_SIZE_BYTES - PICO_FS_DEFAULT_SIZE, 0);
    blockdevice_t *sd = blockdevice_sd_create(spi0,
                                              PICO_DEFAULT_SPI_TX_PIN,
                                              PICO_DEFAULT_SPI_RX_PIN,
                                              PICO_DEFAULT_SPI_SCK_PIN,
                                              PICO_DEFAULT_SPI_CSN_PIN,
                                              24 * MHZ,
                                              true);
    filesystem_t *lfs = filesystem_littlefs_create(500, 16);
    filesystem_t *fat = filesystem_fat_create();

    fs_mount("/", lfs, flash);
    fs_mount("/sd", fat, sd);
    return true;
}

Of course, blockdevice_t and/or filesystem_t objects can also be implemented by users themselves and added to the system.

Program size per library combination

Library Combination	Used size	Region Size	%age Used
SD + Flash + FAT + littlefs	211648 B	2 MB	10.09%
SD + FAT	177776 B	2 MB	8.48%
Flash + FAT	172168 B	2 MB	8.21%
SD + Flash + littlefs	89104 B	2 MB	4.25%
SD + littlefs	88312 B	2 MB	4.21%
Flash + littlefs	82720 B	2 MB	3.94%

Supported POSIX and C standard file API

The pico_filesystem library implements basic Newlib "system calls", allowing users to #include <stdio.h> and use a wide range of POSIX and C standard file APIs. The familiar success is set to 0, failure to -1 and errno. A list of tested APIs is available.

Testing

The pico-filesystem-examples project contains unit and integration tests for pico_filesystem.

This library is BSD-3-Clause licensed. See also the pico-vfs repository for more information.

Regards,

[Memotech-Bill]

It is worth referencing the discussion: https://forums.raspberrypi.com/viewtopic.php?t=368898

[kilograham]

this looks great, thank you.

one initial question from a quick skip read; how does overriding _write play with STDIN/STDOUT/STDERR handles already handled by SDK override of the same? I'm assuming it just breaks it?

I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib), and we should think about how to "inject" mutliple FILE back ends (though arguably once you have a VFS, perhaps it is only the stdin/stdout/stderr that you care about, so the "replacement" APIs could just call back for those

[oyama]

Thank you for your comment,

pico_filesystem VFS linked write(fileno(stdout), ...); returns a 9: Bad file number error or write to a user opened file. the process is passed to the VFS of the pico_filesystem, which is certainly an unintended behaviour of the user. However, calling printf("foo"); does not seem to have any negative effect, as the process is sent to the UART and USB without being _write.
A bad point of the current pico_filesystem VFS is that file descriptors are issued from 0 when a file is opened. This can conflict with STDIN_FILENO/STDOUT_FILENO/STDERR_FINENO of the standard OS and should probably be avoided.

I think pico_stdio_semihosting is more susceptible. However, this also needs to be verified.

[oyama]

A bad point of the current pico_filesystem VFS is that file descriptors are issued from 0 when a file is opened. This can conflict with STDIN_FILENO/STDOUT_FILENO/STDERR_FINENO of the standard OS and should probably be avoided.

This problem has been fixed and the VFS file descriptor for pico_filesystem is now issued from 3. If the user calls write(fileno(stdout), ...) or read(fileno(stdin), ...) , returns -1 and errno == 9.

[oyama]

To avoid inhibiting pico_stdio, file descriptors 0 to 2 are passed to pico_stdio instead of being processed by pico_filesystem.

[kilograham]

I'm putting this in 1.7.0 for now as we are looking at reworking our C library API (so we don't just support newlib)

This is the pico_clib_interface stuff in SDK 2.0.0 ... we should now have a good place to inject FILE support

[oyama]

Thanks for the 2.0.0 release! This is a good hook point. I'll modify it to fit this API and test it.

[oyama]

I finally got my hands on Pico 2 and have made the following fixes and verifications

• Changed PR to conform to pico-sdk 2.0.0
• verified that pico-vfs, the core of the PR implementation, works with Pico/Pico 2 Arm/Pico 2 RISC-V cores

Onboard Flash and SPI SD card operation is also no longer a problem.

[Slion]

This awesome stuff and I'm looking forward using it. Thanks for sharing.

How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten. Do you check and assert if the firmware overlaps with the flash file system specified? Can I check how much space is currently used and how much is available on the file system?

[Slion]

So it looks like the default configuration is not taking care of leaving the Bluetooth storage alone.
https://github.com/oyama/pico-vfs/blob/main/src/filesystem/fs_init.c#L14
It also uses a large chunk of flash and does not try to check if the firmware is small enough.

[Memotech-Bill]

How do you define the range of flash being used by the file system? The end of the flash is notably used by btstack to store paired devices' keys, we don't want those to be overwritten.

It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE.

[Slion]

Does each flash file use at least a 4K sector or is it somehow managed at the flash page level?

Answer:
This is more a question about littlefs really and from the testing I did it looks like littlefs can pack multiple files per sector thankfully.
See: oyama/pico-vfs#60 (comment)

[Slion]

It is possible to change where bluetooth stores its keys by defining PICO_FLASH_BANK_STORAGE_OFFSET and PICO_FLASH_BANK_TOTAL_SIZE.

I'm planning to do it the other way around. I'll define a FS configuration that leaves enough space for my firmware and BT storage.

[Slion]

I got it working fine it seems. For those interested on how it looks for real take a look at that: oyama/pico-vfs#59

Basically the above samples forgot to mention you need to format your file system before it can be mounted and used.

[Slion]

Why do we need the littlefs submodule in pico-sdk when it is already present in pico-vfs?

[Slion]

Worth noting that this also works through C++ fstream.

[oyama]

Thank you for your comment. My response is delayed as I was on a long business trip.

The reason for registering the littlefs submodule in both pico-vfs and pico-sdk is to avoid modifying the pico-sdk installation instructions. As you pointed out, a more natural layout would be to register littlefs as a submodule only once under pico-vfs. However, since git does not clone submodules recursively by default, we would need to adjust the pico-sdk installation process. To avoid this, I added littlefs directly as a submodule in pico-sdk as well, creating a flat submodule structure for the pull request.