CWL version of prefactor
https://github.com/lofar-astron/prefactor/
Note that this project is still very much in alpha stage and very much in flux. It is a prototype implementation and is not intended for end user audience. The goal is to demonstrate various technologies and how they could assist simplifying construction and deployment of big scale data reduction pipelines.
optional:
To run the pipeline with the example dataset just run:
$ make runTo Run using a udocker (user space docker) run:
$ make run-udockerTo try out toil as a schedular run:
$ make toilWe've made a set of ansbile scripts to automate the deployment of a cluster that can run CWL pipelines: https://github.com/EOSC-LOFAR/ansible
If you are on cartesius, you first need to activate a python environment:
$ module load pythonSince all jobs are quite small, it is better to use the staging partition. You can configure toil to use the staging partition with:
$ export TOIL_SLURM_ARGS="-t 0:30:00 -p staging"$ make slurm
the pipeline consists of these steps:
| step name | status | file | comment |
|---|---|---|---|
| createmap_cal | skip | we don't use map files | |
| ndppp_prep_cal | implemented | steps/ndppp_prep_cal.cwl | |
| combine_data_cal_map | skip | We don't use map files | |
| sky_cal | implemented | steps/sky_cal.cwl | |
| sky_cal_path | skip | we don't use map files | |
| calib_cal | implemented | steps/steps/calib_cal.cwl | |
| h5_imp_cal_map | skip | we don't use map files | |
| h5imp_cal | implemented | steps/losoto_importer.cwl | |
| mk_inspect_dir | skip | we don't need to create dirs | |
| copy_h5imp_cal | skip | this is managed by CWL | |
| fitclock | implemented | steps/fitclock.cwl | |
| ampl | implemented | steps/ampl.cwl | disabled for test since not enough subbands |
| plots | implemented | steps/plots.cwl | disabled for test since not enough subbands |
| phase | implemented | steps/phase.cwl | |
| plot_cal_phases | implemented | steps/plot_cal_phases.cwl | |
| createmap_cal_pngs | skip | we don't use map files | |
| copy_cal_pngs | skip | this is managed by CWL | |
| createmap_cal_txts | skip | we don't use map files | |
| copy_cal_txts | skip | this is managed by CWL | |
| mk_cal_values_dir | skip | this is managed by CWL | |
| createmap_cal_npys | skip | we don't use map files | |
| copy_cal_npys | skip | this is managed by CWL |