Why is this bash loop failing to concatenate the files?

Question

I am at my wits end as to why this loop is failing to concatenate the files the way I need it. Basically, lets say we have following files:

AB124661.lane3.R1.fastq.gz
AB124661.lane4.R1.fastq.gz

AB124661.lane3.R2.fastq.gz
AB124661.lane4.R2.fastq.gz

What we want is:

cat AB124661.lane3.R1.fastq.gz AB124661.lane4.R1.fastq.gz > AB124661.R1.fastq.gz
cat AB124661.lane3.R2.fastq.gz AB124661.lane4.R2.fastq.gz > AB124661.R2.fastq.gz

What I tried (and didn't work):

1. Create and save file names (AB124661) to a ID file:

ls -1 R1.gz | awk -F '.' '{print $1}' | sort | uniq > ID

This creates an ID file that stores the samples/files name.

2. Run the following loop:

for i in `cat ./ID`; do cat $i\.lane3.R1.fastq.gz $i\.lane4.R1.fastq.gz \> out/$i\.R1.fastq.gz; done
    
for i in `cat ./ID`; do cat $i\.lane3.R2.fastq.gz $i\.lane4.R2.fastq.gz \> out/$i\.R2.fastq.gz; done

The loop fails and concatenates into empty files.

Things I tried:

• Yes, the ID file is definitely in the folder
• When I run with echo it shows the cat command correct

Any help will be very much appreciated,

Best,

AC

Answer 1

1. why are you escaping the \> ? That's going to result in a cat: '>': No such file or directory instead of a redirection.
2. Don't read lines with for

while IFS= read -r id; do
    cat "${id}.lane3.R1.fastq.gz" "${id}.lane4.R1.fastq.gz" > "out/${id}.R1.fastq.gz"
    cat "${id}.lane3.R2.fastq.gz" "${id}.lane4.R2.fastq.gz" > "out/${id}.R2.fastq.gz"
done < ./ID