compatibility with local and Compute Canada installations (qsub not found)

[andrewcyung]

Hello,

I am trying to install pydpiper and run the test_MBM_and_MAGeT.py script on the downloadable test-data example, but can't get much to work:

1. I tried installing the Singularity container on my local workstation, and cloned the latest version of pydpiper into the container (just so that I could get the testing script. I didn't install pydpiper because it seems to be part of the Singularity container). I then changed sample.cfg queue-type to "local" and removed the "time-to-seppuku" line. The command I use is:

python3 pydpiper_testing/test_MBM_and_MAGeT.py test-data

Unfortunately the program does not seem to move, with the number of stages processed being stuck at zero. Changing the queue-type back to sge comes back with an error that originates from "qsub" not being recognized.

2. I tried installing the Singularity container onto Compute Canada, and ran it with sge as the queue type, but this also seems to fail because of the absence of qsub:

Singularity> python3 pydpiper_testing/test_MBM_and_MAGeT.py test-data

Warning: Not masking your images from atlas masks after LSQ6 alignment ... probably not what you want (this can have negative effects on your registration and statistics) [/usr/local/lib/python3.8/dist-packages/pydpiper/pipelines/MBM.py:319]

Total number of stages in the pipeline:  1858
Number of stages already processed:      0
Daemon is running at: 206.12.124.2:44052
The pipeline's uri is: PYRO:[email protected]:44052
.Traceback (most recent call last):
  File "/usr/local/bin/qbatch", line 8, in <module>
    sys.exit(qbatchParser())
  File "/usr/local/lib/python3.8/dist-packages/qbatch/qbatch.py", line 769, in qbatchParser
    qbatchDriver(**vars(args))
  File "/usr/local/lib/python3.8/dist-packages/qbatch/qbatch.py", line 617, in qbatchDriver
    return_code = subprocess.call(['qsub', script])
  File "/usr/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'qsub'

If I am understanding this correctly, I assume that pydpiper really isn't intended for running on a single workstation, but rather a computing cluster? If so, what are the key requirements for a cluster to be compatible with the use of pydpiper?

Thanks!

[gdevenyi]

Testing, last I checked doesn't work.

#424

Try a real pipeline.