Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fusion symlink resolution doesn't work with directories #4725

Closed
bentsherman opened this issue Feb 8, 2024 · 7 comments · Fixed by #5004
Closed

Fusion symlink resolution doesn't work with directories #4725

bentsherman opened this issue Feb 8, 2024 · 7 comments · Fixed by #5004

Comments

@bentsherman
Copy link
Member

We recently added some logic to handle Fusion symlinks, which happen when an input file is included in a published output. We use the .fusion.symlinks file to detect and resolve any Fusion symlinks when they are published.

However, when publishing a directory, Nextflow does not walk the directory tree and publish each file individually, it just publishes the directory. If Fusion symlinks were staged into this directory, they will not be detected and resolved.

Given a list of input files:

$ ls -1 freshdesk/fd-4463/files/ 
1.txt
2.txt
3.txt
4.txt
5.txt

The following pipeline script will demonstrate this issue:

process AGGREGATE {
  container "quay.io/nextflow/bash"
  publishDir "results", mode: "copy"

  input:
  path(samples), stageAs: 'AnalysisFiles/'

  output:
  path("*")

  script:
  """
  for name in AnalysisFiles/*.txt; do
    touch AnalysisFiles/Analysis_on_\$(basename \${name} .txt)
  done
  """
}

workflow {
    AGGREGATE( files("files/*") )
}

One workaround that I tried is to make the outputs more explicit:

  output:
  path("AnalysisFiles/*.txt", includeInputs: true)
  path("AnalysisFiles/Analysis_on_*")

But this doesn't quite work because of a small bug in the symlink resolution. I will submit a patch so that at least this workaround works.

A more permanent solution would be to walk the directory tree and publish each file explicitly instead of only publishing the directory (#3933). There are other benefits to making this change, see also #3372.

@bentsherman
Copy link
Member Author

Here is another workaround that works with the current version of Nextflow:

process AGGREGATE {
  container "quay.io/nextflow/bash"
  publishDir "results/AnalysisFiles", mode: "copy"

  input:
  path("*.txt")

  output:
  path("*.txt", includeInputs: true)
  path("Analysis_on_*")

  script:
  """
  for name in *.txt; do
    touch Analysis_on_\$(basename \${name} .txt)
  done
  """
}

So there is something to be said for trying to keep the task directory as flat as possible and using the publish mechanism to create the desired directory structure. Still, this may not be possible or easy to do with certain tools, so I would still like to fix it in Nextflow.

@IreneRobles
Copy link

Hi,

I leave this message to encourage finding a solution to the issue.

In our case, we have created a wrapper for a Python pipeline to enable running each step with different computational resources, using Nextflow without modifying the original code. The directory output of each step is used as input for the next step. The folder structure is complex and dependent on various parameters, therefore, the suggested solutions may not solve our problem.

I can always write a bash script that gets the directory path from the trace and places it into the output folder, but this solution is not ideal and I would really like nextflow and fusion to do it for me.

@bentsherman
Copy link
Member Author

Hi Irene, this issue will be solved by #3933 and #4726. Just waiting for the PRs to be approved and merged

@IreneRobles
Copy link

That is fantastic, thank you so much for the update!

@IreneRobles
Copy link

IreneRobles commented Feb 22, 2024

Hi,
I just noticed that my published directories are now being correctly published in the output folder using the fusion system. I think the software must have been patched.

@bentsherman
Copy link
Member Author

Did you implement a workaround?

@IreneRobles
Copy link

IreneRobles commented Feb 22, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants