New config option: publishDir.basePath

[ewels]

New feature

In @nf-core we prefix every publishDir path with params.outdir. For example, to modify the example in the docs:

publishDir "${params.outdir}/data/chunks"

Without this, Nextflow will spit published results into the launch directory, which I think is generally agreed to be a bad idea in most cases (overwriting / ephemeral storage). But to follow best practices of not doing this, the pipeline code needs this arbitrary variable interpolation.

It would be nice to have a new config attribute for publishDir called something like basePath, which would default to the launch directory (existing behaviour) but if set, works for the entire pipeline run.

This would remove quite a lot of boilerplate code and make it easier for people working with non-nf-core pipelines which don't use this convention.

Could also have this as a command-line flag to easily change at run time, similar to -work-dir (-publish-dir?)

Suggest implementation

• Config: publishDir.basePath
• Command line: -publish-dir

If provided, prefix this string to the path in PublishDir.groovy

[bentsherman]

Related to #2661

[pditommaso]

This has been solved by #3942. In a nutshell, relative paths are resolved against the path specified via NXF_FILE_ROOT env variable.

[ewels]

There's no way to set this from within the workflow though, right? We need to be able to set this with params.outdir in nf-core workflows.

[pditommaso]

This may be a good candidate for nextflow.defaults planned setting #4186. Tho it should be verified if it's feasible.

[ewels]

Yeah, it's to prefix rather than overwrite 🤔

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[bentsherman]

This should be addressed by the workflow outputs DSL #4784 which removes the need to specify the base publish path many times