Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn when file() matches a collection of files #5507

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/reference/stdlib.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,10 @@ The following functions are available in Nextflow scripts:
: Stop the pipeline execution and return an exit code and optional error message.

`file( filePattern, [options] )`
: Get one or more files from a path or glob pattern. Returns a [Path](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/file/Path.html) or list of Paths if there are multiple files.
: :::{versionchanged} 24.11.0-edge
The `file()` function should be used only to match a single file. Use `files()` to match a collection of files. In the future, `file()` will raise an error if it does not match exactly one file.
:::
: Get one or more files from a file name or glob pattern. Returns a single [Path](#path) if there is one file, or a collection of Paths if there are multiple files.

: The following options are available:

Expand All @@ -232,7 +235,7 @@ The following functions are available in Nextflow scripts:
: See also: {ref}`Channel.fromPath <channel-path>`.

`files( filePattern, [options] )`
: Convenience method for `file()` that always returns a list.
: Get a collection of files from a file name or glob pattern. Always returns a collection of files. Supports the same options as `file()`.

`groupKey( key, size )`
: Create a grouping key to use with the {ref}`operator-grouptuple` operator.
Expand Down
26 changes: 15 additions & 11 deletions modules/nextflow/src/main/groovy/nextflow/Nextflow.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,7 @@ class Nextflow {

private static final Random random = new Random()


static private fileNamePattern( FilePatternSplitter splitter, Map opts ) {

private static List<Path> fileNamePattern( FilePatternSplitter splitter, Map opts ) {
final scheme = splitter.scheme
final target = scheme ? "$scheme://$splitter.parent" : splitter.parent
final folder = toCanonicalPath(target)
Expand All @@ -68,15 +66,14 @@ class Nextflow {
if( opts == null ) opts = [:]
if( !opts.type ) opts.type = 'file'

def result = new LinkedList()
def result = new LinkedList<Path>()
try {
FileHelper.visitFiles(opts, folder, pattern) { Path it -> result.add(it) }
}
catch (NoSuchFileException e) {
log.debug "No such file or directory: $folder -- Skipping visit"
}
return result

}

static private String str0(value) {
Expand Down Expand Up @@ -104,11 +101,16 @@ class Nextflow {
* @param path A file path eventually including a glob pattern e.g. /some/path/file*.txt
* @return An instance of {@link Path} when a single file is matched or a list of {@link Path}s
*/
static file( Map options = null, def filePattern ) {

static file(Map options = null, def filePattern) {
if( !filePattern )
throw new IllegalArgumentException("Argument of `file` function cannot be ${filePattern==null?'null':'empty'}")
throw new IllegalArgumentException("Argument of `file()` function cannot be ${filePattern==null?'null':'empty'}")
final result = file0(options, filePattern)
if( result instanceof Collection && result.size() != 1 )
log.warn "The `file()` function was called with a glob pattern that matched a collection of files -- use `files()` instead. The `file()` function should only be used to retrieve a single file. This warning will become an error in the future."
return result
}

private static file0( Map options = null, def filePattern ) {
final path = filePattern as Path
final glob = options?.containsKey('glob') ? options.glob as boolean : isGlobAllowed(path)
if( !glob ) {
Expand All @@ -127,9 +129,11 @@ class Nextflow {
return fileNamePattern(splitter, options)
}

static files( Map options=null, def path ) {
def result = file(options, path)
return result instanceof List ? result : [result]
static Collection<Path> files(Map options=null, def filePattern) {
if( !filePattern )
throw new IllegalArgumentException("Argument of `files()` function cannot be ${filePattern==null?'null':'empty'}")
final result = file0(options, filePattern)
return result instanceof Collection ? result : [result]
}


Expand Down
4 changes: 2 additions & 2 deletions modules/nextflow/src/test/groovy/nextflow/NextflowTest.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -74,13 +74,13 @@ class NextflowTest extends Specification {
Nextflow.file(null)
then:
e = thrown(IllegalArgumentException)
e.message == 'Argument of `file` function cannot be null'
e.message == 'Argument of `file()` function cannot be null'

when:
Nextflow.file('')
then:
e = thrown(IllegalArgumentException)
e.message == 'Argument of `file` function cannot be empty'
e.message == 'Argument of `file()` function cannot be empty'
}

def 'should return http path' () {
Expand Down