module extensions are not included in output of ml --terse avail

[boegel]

Describe the bug

Module extensions show up in the output of ml avail:

$ ml avail numpy/

-------------------------------------------------------------------------------------------------- This is a list of module extensions "module --nx avail ..." to not show.
 ---------------------------------------------------------------------------------------------------
    numpy (E)

But they are not included in the output of ml --terse avail:

$ ml --terse avail numpy/
$

I guess the reason could be that the output should somehow discriminate between actual modules and extensions?

Maybe an additional option like --with-extensions can be added to opt-in to also getting extensions listed, something like:

$ ml --terse --with-extensions avail numpy/
numpy (E)

Note also that the This is a list of module line is excessive long, it doesn't seem to take into account the width of the terminal like module avail does do for modules to avoid line wrapping of the "header line" (---- ... ----)

[rtmclay]

The module extensions cannot be output like modules because they are not modules. The output of "module --terse avail" is used for tab completion. So you can't do "module load numpy" if numpy is an extension.

What purpose would including module extensions serve when doing "module --terse avail". How would anyone or any program know the difference between a module and a module extension?

Any output of module extensions would have to be different from modules. Maybe:

python3/
python3/3.10.1
^python3/3.10.1:numpy
^python3/3.10.1:scipy
git/
git/2.41.0

where only python3/3.10.1 and git/2.41.0 are modules

[rtmclay]

Or just

python3/
python3/3.10.1
^numpy
^scipy
git/
git/2.41.0

Since module avail only prints the extensions and not where they are from.

[rtmclay]

There is a good reason to NOT provide extensions when doing "module --terse avail". Currently, Lmod does not need to build the spider cache when doing "module --terse avail". To print out the extension with --terse, would require building the spider cache. This is an expense, I don't want everyone to pay. Since "module --terse avail" is use for tab completion, we don't want to incur that time penalty for every time during "module load ".

[rtmclay]

The testing branch has been merged to the main branch and has been released as Lmod 8.7.36

[boegel]

With Lmod 8.7.37, I see there's some progress on this.

With a module file matplotlib/3.8.2-gfbf-2023b.lua that includes this:

if convertToCanonical(LmodVersion()) >= convertToCanonical("8.2.8") then
    extensions("contourpy/1.2.0,Cycler/0.12.1,fonttools/4.47.0,kiwisolver/1.4.5,matplotlib/3.8.2")
end

I see this at the end of module --terse_show_extensions avail:

#    Cycler
#    contourpy
#    fonttools
#    kiwisolver
#    matplotlib

A coupe of questions/observations here:

• The version info is stripped out, is that deliberate?
• There's no way of telling in which module(s) these extensions are available?
• $ ml --terse_show_extensions avail does not work, I get:

  Option: "--terse_show_extensions" is unknown.
  Try ml --help for usage.
  

  ml --terse avail works fine though, is that deliberate?

[boegel]

As discussed during today's Lmod monthly meeting: the current implementation of --terse_show_extensions (which should also work for ml, but doesn't yet) is a terse (machine-readable way) of getting the same information that module avail provides. In that sense, it makes sense that there no version information for extensions, and that there no way to determine in which modules an extension is available (that's what spider is for).

That said, it would be great if both the version info and relation to module files could be made available somehow in terse mode as well. Maybe through an additional option like --terse_show_extensions_detailed?

[boegel]

There's also no help text for the --terse_show_extensions option:

$ module  --help 2>&1 | grep 'terse_show_extensions'
  --terse_show_extensions

[rtmclay]

The Lmod branch IS667-terse-av-ext has been pushed to github. This version fixes all known issues. The output just reports the individual extensions instead of at the end. Here is typical output:

/opt/apps/modulefiles/Core:
snake/
snake/1.0
#    A/1.0
#    B/2.0
#    CC/3.0
snake/2.0
#    A/2.0
#    B/3.0
#    CC/3.5
tex/
tex/2010

Please test this branch to see if it works for you.

[boegel]

I took this for a quick spin, and it's working as expected for me.

Here's the relevant bit of the output for a SciPy-bundle module that includes a bunch of extensions:

$ ml --terse_show_extensions avail
...
SciPy-bundle/2023.07-iimkl-2023a
#    beniget/0.4.1
#    Bottleneck/1.3.7
#    deap/1.4.0
#    gast/0.5.4
#    mpmath/1.3.0
#    numexpr/2.8.4
#    numpy/1.25.1
#    pandas/2.0.3
#    ply/3.11
#    pythran/0.13.1
#    scipy/1.11.1
#    tzdata/2023.3
#    versioneer/0.29

This makes it clear which version of the extensions are available, and which module they are available in.

So lgtm!

[rtmclay]

This branch has been merged onto the main branch and has been released as Lmod 8.7.48. Closing this issue.