storage latency features

[colindaven]

Hi,

our environment still suffers from latency (including but not limited to NFS storage) which I can only seem to deal with by introducing sleep 300 commands after data intensive commands in nextflow processes. Think long read filtering of 60 GB input, or outputting an assembly with over 20 GB of files.

Snakemake has the following commands to deal with latency. I haven't seen any of this in Nextflow so far (am I missing something?). Is it planned or maybe even not needed for some reason ?

I think the --latency-wait might be useful for me and some others, see 151 errors at #4272

Thanks

--latency-wait, --output-wait, -w
Wait given seconds if an output file of a job is not present after the job finished. This helps if your filesystem suffers from latency (default 5).

Default: 5

--wait-for-files
Wait –latency-wait seconds for these files to be present before executing the workflow. This option is used internally to handle filesystem latency in cluster environments.

--wait-for-files-file
Same behaviour as –wait-for-files, but file list is stored in file instead of being passed on the commandline. This is useful when the list of files is too long to be passed on the commandline.

--queue-input-wait-time
Set the interval in seconds to check for new input in rules that use from_queue to obtain input files.

Default: 10