Tracking issue regarding changes required to support new modules

[xynydev]

We have discussed the usage of Nushell for developing modules here: blue-build/modules#212
We're working on a v2 of the default-flatpaks module here: blue-build/modules#336

Here's a quick list of things that need to be implemented in CLI to finish off the support for the new types of modules and default-flatpaks v2:

• Support for modules published as separate images under the source image path
  • See https://github.com/blue-build/modules/pkgs/container/modules%2Fdefault-flatpaks
• Support for specifying the version of the module to use (corresponding to the image tag)
  • Proposed syntax: type: default-flatpaks@v1, type: default-flatpaks@v2, type: default-flatpaks@latest etc.
  • Fetch @latest by default
  • Needs schema changes
• Support for module scripts without .sh suffix
  • A script in Nushell should not use the .sh suffix.
  • Need to figure out how to handle existing modules, including custom modules
  • Ideas:
    1. Expect .sh to be there by default, unless the module.yml declares that the module is in Nushell
    2. Expect .sh to not be there, rename the script files of all modules (would break builds with earlier versions, but could be mitigated with symlinks) (local modules might need a different procedure)
    3. Run the Nushell module with a bash wrapper (using an .sh file that is a shim for the .nu script) (no CLI changes required)
• Support for modules written in Nushell
  • https://github.com/blue-build/nushell-image shall be used as the distribution mechanism for Nushell
    • Ex. COPY --from=ghcr.io/blue-build/nushell-image:0.98.0 /nu/nu /usr/bin/nu
  • Ideally the supported Nu version is declared by the repository (or possibly even the module)
  • But it would be wasteful to ship multiple Nu binaries in the images
  • It would be wasteful to ship a Nu binary at all if the module does not have any post-boot elements
  • Ideas:
    1. Modules using Nushell declare the version the want in their module.yml, that is read by the CLI, and that image is bind-mounted to their RUN-step. If a module needs Nushell post-boot, it can cp the binary to the appropriate location.
    2. Modules using Nushell declare the version the want in their module.yml, that is read in the build process for the module repository, and the binaries are added into the image for the module alongside the source code. If a module needs Nushell post-boot, it can cp the binary to the appropriate location.
    3. Nushell replaces yq (and possibly others) and goes alongside bluebuild as one of the binaries that gets included in every image. The version is the same across all builds and modules (need to make sure all modules support the newer version before upgrading, but wasting resources on shipping multiple versions becomes impossible).

[gmpinder]

So if we're wanting to have a way to declare the version of NuShell used for the module, this might require having the binaries mounted for version and module call. We could then have the run_module script set the PATH variable with the mounted path for the nushell bins.

If the property for the nushell version doesn't exist, we could assume either:

• No NuShell binary image is mounted
• We default to the latest version

If we're going to go the route of allowing NuShell version specification, I don't think we would be able to ship images with a NuShell binary as they could be very different versions used. Which would mean that for runtime modules (modules that run when the user is booted), we should rely soley on using Bash since we can pretty much guarantee that shell will be available.

On another note, I think for ease of use we should have a label on the image containing the contents of module.yml as JSON so that the CLI can have an easier time getting that information for each versioned image.

[xynydev]

If we're going to go the route of allowing NuShell version specification, I don't think we would be able to ship images with a NuShell binary as they could be very different versions used.

Could we set the version to be same for the whole repo? Keeping it updated while slightly controlling it such that the builds don't break would be ideal. Not having to re-release the CLI on every version change would be nice too.

Maybe a recipe parameter to set the Nushell version tag in the same vein as we can set the CLI version from the recipe. The default could be a special tag in our Nushell repo that points to the currently supported version.

[gmpinder]

Could we set the version to be same for the whole repo? Keeping it updated while slightly controlling it such that the builds don't break would be ideal. Not having to re-release the CLI on every version change would be nice too.

I don't think that would work well with keeping versioned modules. If we ended up making a v2 of a module and started using a new version of nu, anyone using the v1 module could end up running into an issue where the code for that version is using features that are different/no longer exist in the new nu version.

Maybe a recipe parameter to set the Nushell version tag in the same vein as we can set the CLI version from the recipe. The default could be a special tag in our Nushell repo that points to the currently supported version.

I'd like to keep most maintenance burdens away from the user. They shouldn't have to manage which version of nushell they need to use for the different versions of modules. Remember the goal is to try to make image generation easy for users.