Code

Automatic task cleanup#3849

Code

Draft

bentsherman wants to merge 40 commits intomasternextflow-io/nextflow:masterfrom 452-eager-cleanupnextflow-io/nextflow:452-eager-cleanupCopy head branch name to clipboard

+816-141

Commits

Commits on Mar 27, 2023

• Add initial task graph and metadata json file

  Show description for 47d0168
  bentsherman

  committed
  47d0168

Commits on Mar 28, 2023

• Add task inputs and outputs to conrete DAG

  Show description for ae67027
  bentsherman

  committed
  ae67027

• Fix failing tests

  Show description for 8f95cd6
  bentsherman

  committed
  8f95cd6

Commits on Mar 29, 2023

• Use path-based APIs to get file metadata

  Show description for 9f11e4b
  bentsherman

  committed
  9f11e4b

• Merge branch 'master' into ben-task-graph

  Show description for db6aed1
  bentsherman

  authored
  db6aed1

Commits on Mar 30, 2023

• Use buffer to compute checksum

  Show description for 8456892
  bentsherman

  committed
  8456892

Commits on Mar 31, 2023

• Add support for temporary output paths

  Show description for 77f2cdc
  bentsherman

  committed
  77f2cdc

• Fix failing test

  Show description for 3e55ad5
  bentsherman

  committed
  3e55ad5

• Add caveat about overlapping output channels [ci skip]

  Show description for e307f75
  bentsherman

  committed
  e307f75

Commits on Apr 1, 2023

• Delete files instead of emptying them (now supports directories and remote paths)

  Show description for 08881b0
  bentsherman

  committed
  08881b0

Commits on Apr 10, 2023

• Add `eager' cleanup option

  Show description for 0cf07ec
  bentsherman

  committed
  0cf07ec

• Fix concurrency issues [ci fast]

  Show description for 73b2f3b
  bentsherman

  committed
  73b2f3b

Commits on Apr 21, 2023

• Merge branch 'master' into ben-task-graph-pull

  bentsherman

  committed
  0dd98d6

Commits on Apr 26, 2023

• Merge branch 'master' into ben-task-graph-pull

  Show description for 0f505d3
  bentsherman

  committed
  0f505d3

• Replace synchronized with lock

  Show description for e81e584
  bentsherman

  committed
  e81e584

Commits on Apr 28, 2023

• Merge branch 'ben-task-graph-pull' into 452-eager-cleanup

  Show description for ba2e7a6
  bentsherman

  committed
  ba2e7a6

• Remove dependency on task graph branch

  Show description for f7bcfa8
  bentsherman

  committed
  f7bcfa8

• Use downstream tasks to determine lifetime for task cleanup

  Show description for 4d90e27
  bentsherman

  committed
  4d90e27

Commits on Jul 7, 2023

• Rename TemporaryFileObserver to TaskCleanupObserver

  Show description for dd23b2a
  bentsherman

  committed
  dd23b2a

• Merge branch 'master' into 452-eager-cleanup

  Show description for 6a34be6
  bentsherman

  committed
  6a34be6

• Wait for output files to be published

  Show description for ff08984
  bentsherman

  committed
  ff08984

• Log warning if eager cleanup is used with incompatible publish modes

  Show description for 9b343b6
  bentsherman

  committed
  9b343b6

• Add eager cleanup for individual output files

  Show description for 6b5a820
  bentsherman

  committed
  6b5a820

Commits on Jul 10, 2023

• Add warning about includeInputs with eager cleanup

  Show description for c42f249
  bentsherman

  committed
  c42f249

Commits on Jul 11, 2023

• Add resumability

  Show description for 8b66c39
  bentsherman

  committed
  8b66c39

• Upgrade kryo to 5.4.0 (#3562)

  Show description for 43c939d
  bentsherman

  committed
  43c939d

• Add serializer for HashCode

  Show description for 5a78cf9
  bentsherman

  committed
  5a78cf9

• Fix failing tests

  Show description for 7b02b8e
  bentsherman

  committed
  7b02b8e

Commits on Jul 12, 2023

• Revert Kryo upgrade

  Show description for c727c05
  bentsherman

  committed
  c727c05

• Add kryo instantiator for HashCode

  Show description for d37ee1f
  bentsherman

  committed
  d37ee1f

• Remove HashCode serializer

  Show description for 859e96b
  bentsherman

  committed
  859e96b

• Add log messages, fix infinite hang

  Show description for c89811c
  bentsherman

  committed
  c89811c

• Add support for includeInputs

  Show description for a034f60
  bentsherman

  committed
  a034f60

• Update log messages

  Show description for 9741b1c
  bentsherman

  committed
  9741b1c

• Add lazy, eager, aggressive cleanup strategies

  Show description for 3aae1af
  bentsherman

  committed
  3aae1af

Commits on Sep 13, 2023

• Merge branch 'master' into 452-eager-cleanup

  Show description for 08342dd
  bentsherman

  committed
  08342dd

Commits on Sep 14, 2023

• Add thread pool for task cleanup

  Show description for 3498c5d
  bentsherman

  committed
  3498c5d

• Minor improvement to checkCachedOutput()

  Show description for c421617
  bentsherman

  committed
  c421617

Commits on Jan 31, 2024

• Merge branch 'master' into 452-eager-cleanup

  Show description for 5b05aad
  bentsherman

  committed
  5b05aad

Commits on Feb 1, 2024

• minor edits

  Show description for 9ce10dc
  bentsherman

  committed
  9ce10dc