Overwrite published outputs only if they are stale

[bentsherman]

Close #3372

This PR builds on #3933 , which ensures that files in a directory are explicitly published. Given this behavior, we can then compare the source and target checksums to decide whether to publish a file.

• If the checksums match then the copy can be skipped
• If the checksums don't match, then the target should always be overwritten, even if the task was cached, because if the previous run was terminated and publishing cancelled, the published file might be incomplete.

The checksum code is ripped from #3802 , as it is also needed for task provenance and the resumability for automatic cleanup (#3849). It tries to use a pre-computed checksum (such as ETag for cloud storage paths) where possible to avoid scanning the file contents.

The only qualm I have right now is that the checksum code doesn't use the ETag for Google cloud storage paths. This is because we use the built-in file system provider from the Google SDK rather than our own. Maybe we can just extend this FS provider and register it over the existing one, but I'm not sure yet how to do it.

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	9edc796
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/664bbce9dfbe6900083d22f7
😎 Deploy Preview	https://deploy-preview-4729--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[bentsherman]

I added a custom NIO filesystem for Google cloud storage which just delegates to the existing one, but allows us to add custom behavior like this ETag support.

It turns out that the ETag returned by the Java SDK is a CRC32 instead of MD5. The MD5 is also available through this storage API, but I haven't gotten it to work yet. The metadata is always null, even though I provided the projectId and I have the credentials file. Need to investigate further when I have more time

[bentsherman]

Some notes from discussion today (related to workflow output DSL):

• checksum verification should be optional as it can be very expensive for non-object storage
• should also have a lightweight alternative that simply checks the last modified timestamp and file size (similar to cache directive)

[bentsherman]

This PR is mostly covered by the workflow output definition #4784 . The only remaining piece is the ETagAware interface which uses object metadata instead of computing the checksum, which makes the deep overwrite mode more feasible.

I propose that we merge that piece and leave out the Google NIO filesystem for now since I never quite got it to work. I can finish it in a separate PR. Then we can move this PR forward for S3 and Azure.