micromamba-exported explicit conda lock files don't work over http, or named as .lock

[pinin4fjords]

Bug report

Expected behavior and actual behavior

Expected:

• If a Conda lock file works locally, we should be able to supply it by http.
• We should also have clear documentation around required naming/ file extensions for lock files.

Observed

Here is a conda lockfile produced with micromamba like micromamba env export --explicit > env.txt:

env.txt

This works:

#!/usr/bin/env nextflow

process FOO {

    debug true
    conda 'env.txt'

    script:
    """
    R --version
    """

}

workflow {
    FOO()
}

We would expect the same file hosted remotely to work:

...
conda 'https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'
...

... but it does not:

Nextflow 24.09.2-edge is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [cheesy_marconi] DSL2 - revision: 510b3a1397

[-        ] FOO -
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a]
ERROR ~ Error executing process > 'FOO'

Caused by:
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-eeddd0960df503db901cc9c3644e607a --file /workspace/gitpod/hello-nextflow/foo/https\:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    
    CondaFileIOError: '/workspace/gitpod/hello-nextflow/foo/https:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'. [Errno 2] No such file or directory: '/workspace/gitpod/hello-nextflow/foo/https:/gist.githubusercontent.com/pinin4fjords/e8cdca9521586f667ed1c5bcd33598d9/raw/550f0f4ed17f5d2741552a2b9f1d625239e499ec/env.txt'



 -- Check '.nextflow.log' file for details

Naming the file with .lock, even locally does not work:

...
    conda 'env.lock'
...

(i.e renaming env.txt to env.lock):

Nextflow 24.09.2-edge is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [awesome_dalembert] DSL2 - revision: 021eb262d1

[-        ] FOO -
Creating env using conda: env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242 env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
[-        ] FOO -
Creating env using conda: env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-f187ecbc64371e10748465ebfbff2242 env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): ...working... done
    Solving environment: ...working... failed
    
    PackagesNotFoundError: The following packages are not available from current channels:
    
      - env.lock
    
    Current channels:
    
      - https://conda.anaconda.org/conda-forge
      - https://conda.anaconda.org/bioconda
      - defaults
      - https://repo.anaconda.com/pkgs/main
      - https://repo.anaconda.com/pkgs/r
    
    To search for alternate channels that may provide the conda package you're
    looking for, navigate to
    
        https://anaconda.org
    
    and use the search bar at the top of the page.



 -- Check '.nextflow.log' file for details

Neither does supplying the .lock file by http:

...
    conda 'https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock'
...

Error:

 N E X T F L O W   ~  version 24.02.0-edge

 ┃ Launching `main.nf` [nasty_torvalds] DSL2 - revision: b01a400c6a

[-        ] FOO -
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4 https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
[-        ] FOO -
Creating env using conda: https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock [cache /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4]
ERROR ~ Error executing process > 'FOO'

Caused by:
  Failed to create Conda environment
  command: conda create --mkdir --yes --quiet --prefix /workspace/gitpod/hello-nextflow/foo/work/conda/env-ea0ebf23bfe548bbed42e257c0d75fa4 https://gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
  status : 1
  message:
    /opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py:248: FutureWarning: `--mkdir` is deprecated and will be removed in 25.3. Redundant argument.
      super().__call__(parser, namespace, values, option_string)
    Channels:
     - conda-forge
     - bioconda
     - defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): ...working... done
    Solving environment: ...working... failed
    
    PackagesNotFoundError: The following packages are not available from current channels:
    
      - //gist.githubusercontent.com/pinin4fjords/c5a32673f084cdb772b03a63b9b1d118/raw/c6493bad0819bf8dff0c342af06eaeeee9c03af0/env.lock
    
    Current channels:
    
      - https://conda.anaconda.org/conda-forge
      - https://conda.anaconda.org/bioconda
      - defaults
      - https://repo.anaconda.com/pkgs/main
      - https://repo.anaconda.com/pkgs/r
    
    To search for alternate channels that may provide the conda package you're
    looking for, navigate to
    
        https://anaconda.org
    
    and use the search bar at the top of the page.



 -- Check '.nextflow.log' file for details

Environment

• Nextflow version: 24.09.2-edge
• Java version: [?]
• Operating system: Linux
• Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

[ewels]

I don't think that this is a bug, more just that the behaviour you expected is not supported.

There are two separate things here: handling remote files and handling different file extensions. I've created two new issues to track these separately, so that we can keep discussion focussed. Let's continue there:

• Support remote files for conda directive #5582
• Conda: Support .lock file extensions. #5583