Unclear development requirements

[sdruskat]

Backround: I was cloning this repository to implement podman pull support in cwltool (since done my @mr-c directly 🙏).

Expected Behavior

• Developing on WSL2 with Ubuntu, I expected that after following the instructions in CONTRIBUTING.md, no tests would fail.

• I would expect either

  • CONTRIBUTING.md to include the requirements mentioned below (including how to fix the one test that still fails), or
  • some way of installing the missing test dependencies (docker, udocker, SingularityCE) to be installed automatically through make, and the tests checking for their existence/runtime and failing with a respective error message if requirements aren't met, or
  • tests being skipped when the mentioned requirements aren't met (as I wouldn't want to force developers to create this environment).

Actual Behavior

• After following the instructions in CONTRIBUTING.md, tests failed, and the test run hung at a specific test (test_windows_warning.py).
• After applying the partial solution below, there is still one test failing (test_whoami) with the stacktrace below.

Partial solution:

• Investigation showed that there are undocumented requirements for passing tests locally:
  1. A running Docker daemon. Running the daemon (sudo dockerd) caused some previously failing tests to pass, and the test run to complete (rather than hang at the above-mentioned test.
  2. A local installation of udocker. Installing udocker on the test system caused some previously failing tests to pass.
  3. A specific installation of singularity. Tests failed with apptainer (the Linux Foundation fork of the Singularity codebase that provides the singularity command) installed. Installing SingularityCE 3.10.4 (https://github.com/sylabs/singularity/releases/download/v3.10.4/singularity-ce_3.10.4-focal_amd64.deb) caused some previously failing tests to pass.

Workflow Code

n/a

Full Traceback

(env) stephan@computer:~/src/cwltool$ make test
python --version 2>&1 | grep "Python 3"
Python 3.10.8
python -m pytest -rs
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/stephan/src/cwltool, configfile: tox.ini, testpaths: tests
plugins: xdist-3.0.2, mock-3.10.0, cov-4.0.0, httpserver-1.0.6
gw0 [642] / gw1 [642]
............................................................................................................................................................................ssss................ss....... [ 31%]
...............................................................................................................................s.........s............................................................... [ 62%]
..........................................................................................................................................................F.....s....................................s... [ 93%]
.......................................                                                                                                                                                                   [100%]sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=0 mode='r' encoding='UTF-8'>
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=1 mode='w' encoding='UTF-8'>

=================================================================================================== FAILURES ====================================================================================================
__________________________________________________________________________________________________ test_whoami __________________________________________________________________________________________________
[gw0] linux -- Python 3.10.8 /home/stephan/src/cwltool/env/bin/python

    def test_whoami() -> None:
        username, fullname = provenance._whoami()
        assert username and isinstance(username, str)
>       assert fullname and isinstance(fullname, str)
E       AssertionError: assert ('')

tests/test_provenance.py:765: AssertionError
=============================================================================================== warnings summary ================================================================================================
env/lib/python3.10/site-packages/galaxy/util/path/__init__.py:5
env/lib/python3.10/site-packages/galaxy/util/path/__init__.py:5
  /home/stephan/src/cwltool/env/lib/python3.10/site-packages/galaxy/util/path/__init__.py:5: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
    import imp

tests/test_path_checks.py:110
tests/test_path_checks.py:110
  /home/stephan/src/cwltool/tests/test_path_checks.py:110: PytestCollectionWarning: cannot collect test class 'TestFsAccess' because it has a __init__ constructor (from: tests/test_path_checks.py)
    class TestFsAccess(StdFsAccess):

tests/test_dependencies.py::test_biocontainers
tests/test_dependencies.py::test_biocontainers_resolution
  /home/stephan/src/cwltool/env/lib/python3.10/site-packages/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
    warnings.warn(

tests/test_examples.py::test_js_console_cmd_line_tool[]
  /home/stephan/src/cwltool/env/lib/python3.10/site-packages/cwl_utils/sandboxjs.py:453: FutureWarning: Possible nested set at position 2
    re.findall(r"^[[](?:log|err)[]].*$", stderr, flags=re.MULTILINE)

tests/test_provenance.py::test_secondary_files_output
tests/test_provenance.py::test_directory_workflow
tests/test_provenance.py::test_no_data_files
tests/test_provenance.py::test_hello_workflow
tests/test_provenance.py::test_hello_single_tool
tests/test_provenance.py::test_revsort_workflow
tests/test_provenance.py::test_nested_workflow
tests/test_provenance.py::test_secondary_files_implicit
tests/test_provenance.py::test_secondary_files_explicit
  /home/stephan/src/cwltool/env/lib/python3.10/site-packages/rdflib/plugins/serializers/nt.py:35: UserWarning: NTSerializer always uses UTF-8 encoding. Given encoding was: None
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================ short test summary info ============================================================================================
SKIPPED [1] tests/test_cuda.py:25: nvidia-smi required for CUDA not detected
SKIPPED [1] tests/test_cuda.py:37: nvidia-smi required for CUDA not detected
SKIPPED [1] tests/test_cuda.py:50: nvidia-smi required for CUDA not detected
SKIPPED [1] tests/test_cuda.py:61: nvidia-smi required for CUDA not detected
SKIPPED [1] tests/test_dependencies.py:57: modulecmd not installed
SKIPPED [1] tests/test_dependencies.py:79: modulecmd not installed
SKIPPED [1] tests/test_ext.py:76: This is not the default behaviour yet
SKIPPED [1] tests/test_ext.py:239: This test is non-deterministic
SKIPPED [1] tests/test_singularity.py:18: Requires that version 2.6.x of singularity executable version is on the system path.
SKIPPED [1] tests/test_singularity.py:110: Requires that version 2.6.x of singularity executable version is on the system path.
====================================================================== 1 failed, 631 passed, 10 skipped, 16 warnings in 349.68s (0:05:49) =======================================================================
make: *** [Makefile:162: test] Error 1

Your Environment

• OS:

WSL2, kernel version 5.10.16, with OS 👇

LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.5 LTS
Release:        20.04
Codename:       focal

• cwltool version: 3.1.20221125131128

[mr-c]

Thank you @sdruskat ; For the docker issues, it seems you had docker installed, but not running. Perhaps your WSL setup didn't match https://github.com/common-workflow-language/cwltool#ms-windows-users ?

I'm surprised to hear about udocker; the tests install it automatically in a temporary directory. Perhaps udocker doesn't work under WSL2?

Which Singularity version were you running? Does Singularity even work with WSL2?

[mr-c]

• After applying the partial solution below, there is still one test failing (test_whoami) with the stacktrace below.

Thanks, does #1771 fix that test for you?

[sdruskat]

Thank you @sdruskat ; For the docker issues, it seems you had docker installed, but not running.

Yes. Installed in a way that doesn't start the daemon on startup. And it wasn't clear from the contribution guidelines that it should be running. Perhaps a link to the general setup section README would be helpful, although the text there still doesn't cater for setups where dockerd is started manually.

Perhaps your WSL setup didn't match https://github.com/common-workflow-language/cwltool#ms-windows-users ?

It did match, but the text marks docker et al. as optional, so there's not necessarily a link to the development guidelines.

I'm surprised to hear about udocker; the tests install it automatically in a temporary directory. Perhaps udocker doesn't work under WSL2?

It does (installed from released version following https://indigo-dc.github.io/udocker/installation_manual.html), and the tests passed after doing this 🤷.

Which Singularity version were you running? Does Singularity even work with WSL2?

Initially apptainer 1.3.1, removing it and installing SingularityCE fixed this (I think with apptainer, singularity --version would return the apptainer version 1.3.1, which tests check for).

[sdruskat]

• After applying the partial solution below, there is still one test failing (test_whoami) with the stacktrace below.

Thanks, does #1771 fix that test for you?

It does indeed 🎉.