Nextflow name collision

Question

I have files with identical names but in different folders. Nextflow stages these files into the same work directory resulting in name collisions. My question is how to deal with that without renaming the files. Example:

# Example data
mkdir folder1 folder2
echo 1 > folder1/file.txt
echo 2 > folder2/file.txt

# We read from samplesheet
$ cat samplesheet.csv
sample,file
sample1,/home/atpoint/foo/folder1/file.txt
sample1,/home/atpoint/foo/folder2/file.txt

# Nextflow main.nf
#! /usr/bin/env nextflow

nextflow.enable.dsl=2

// Read samplesheet and group files by sample (first column)
samplesheet = Channel
    .fromPath(params.samplesheet)
    .splitCsv(header:true)
    .map {
            sample = it['sample']
            file   = it['file']
            tuple(sample, file)
}
        
ch_samplesheet = samplesheet.groupTuple(by:0)

// That creates a tuple like:
// [sample1, [/home/atpoint/foo/folder1/file.txt, /home/atpoint/foo/folder2/file.txt]]

// Dummy process that stages both files into the same work directory folder
process PRO {

    input:
    tuple val(samplename), path(files)

    output:
    path("out.txt")

    script:
    """
    echo $samplename with files $files > out.txt
    """

}

workflow { PRO(ch_samplesheet) }

# Run it
NXF_VER=21.10.6 nextflow run main.nf --samplesheet $(realpath samplesheet.csv)

...obviously resulting in:

N E X T F L O W  ~  version 21.10.6
Launching `main.nf` [adoring_jennings] - revision: 87f26fa90b
[-        ] process > PRO -
Error executing process > 'PRO (1)'

Caused by:
  Process `PRO` input file name collision -- There are multiple input files for each of the following file names: file.txt

So, what now? The real world application here is sequencing replicates of the same fastq file, which then have the same name, but are in different folders, and I want to feed them into a process that merges them. I am aware of this section in the docs but cannot say that any of it was helpful or that I understand it properly.

Answer 1

You can use stageAs option in your process definition.

#! /usr/bin/env nextflow
nextflow.enable.dsl=2

samplesheet = Channel
    .fromPath(params.samplesheet)
    .splitCsv(header:true)
    .map {
        sample = it['sample']
        file = it['file']
        tuple(sample, file)
     }
    .groupTuple()
    .set { ch_samplesheet }

// [sample1, [/path/to/folder1/file.txt, /path/to/folder2/file.txt]]

process PRO {
    input:
        tuple val(samplename), path(files, stageAs: "?/*")

    output:
        path("out.txt")

    shell:
        def input_str = files instanceof List ? files.join(" ") : files
        """
        cat ${input_str} > out.txt
        """
}

workflow { PRO(ch_samplesheet) }

See an example from nf-core and the path input type docs