Deleting all work directories, if they were not necesarry for the most recent pipeline run

[photocyte]

Hi there,

Wondering, is there a way to delete all the specific work directories that were not CACHED for -resume, or otherwise contained files used or produced by the most recent pipeline execution?

The rationale is, I develop Nextflow pipelines by running them a bunch of times, while editing the script. This means that lineages of previously CACHED results, might no longer be valid due to editing of a early process script, so these results end up orphaned & just taking up space.

The simplest way to clean this up is just to delete the whole work directory, and rerun the pipeline from scratch, but that requires potentially rerunning very long tasks that had valid CACHED results. So instead, I'd like to delete those unnecessary directories, while also leaving my current CACHED results fully intact. In other words, this would be a way to reduce the work directory to the same net result of # of folders & files, as a clean run of the pipeline. I assume the Nextflow understanding of its own execution DAG/previous results, is sufficient to sort out which folders to keep and remove.

I haven't seen anything in the Nextflow documentation about this (i.e. cleanup = true in nextflow.config, https://www.nextflow.io/docs/latest/config.html#miscellaneous not what I'm looking for). I've also read through this thread: #452 and don't think the mentioned clean_work_dirs.sh is the solution either, although it is a neat trick.

Thoughts?

All the best,
-Tim

[bentsherman]

As I understand it, you want to do something like:

1. Run a pipeline
2. Change a few things in the pipeline
3. Resume the pipeline
4. Automatically delete any task directories that were not cached by the resumed run

I don't think it would make sense for Nextflow to do this automatically, because it is common for a work directory to contain tasks from many different runs. But, I think you could do this yourself with the nextflow log command and some Bash.

nextflow log prints the list of all previous runs. nextflow log last by default prints the task directories of the most recent run. Using this command, you could then delete all task directories that aren't in that list. Then put all of that in a wrapper script around Nextflow and you're good to go. 😄

[photocyte]

Thanks! I made a nextflow_cleanup_extraneous_work.sh gist to do this.

I don't think the strategy would exactly work, as there are tmp directories that are deleted by the script and presumably conda dependencies that could be installed as well (not present in this case) could be deleted. But works for my purposes.

[bentsherman]

Nicely done. The work/tmp stuff should be minimal. You can preserve the conda cache by setting conda.cacheDir or NXF_CONDA_CACHEDIR to a different directory. I do that anyway to keep a central cache.

[Gullumluvl]

@photocyte very useful script thanks!

@bentsherman :

I don't think it would make sense for Nextflow to do this automatically, because it is common for a work directory to contain tasks from many different runs. But, I think you could do this yourself with the nextflow log command and some Bash.

Hi, I have a short experience so I am only seeing it from my narrow point of view, but I would argue for a way to do this with Nextflow: first it would be safer than a custom-made bash script. It might also be able to take into account the cached collected files, that do not show up in nextflow log. Also, it would be even cooler if it could take into account the files that were output (e.g. with publishDir and similar directives), because those are symlinks by default.

If people want to keep multiple runs (e.g. different parametrizations of their analysis? different workflows?), what stops us from listing and preserving multiple runs instead of just the last? (the user would have to save a list of the "meaningful" runs somewhere but that's it).