How to run compression in gnu parallel?

Question

Hi I am trying to compress a file with the bgzip command

bgzip -c 001DD.txt > 001DD.txt.gz

I want to run this command in parallel. I tried:

parallel ::: bgzip -c 001DD.txt > 001DD.txt.gz

but it gives me this error:

parallel: Error: Cannot open input file 'bgzip': No such file or directory

Answer 1

You need to chop the big file into smaller chunks and compress these. It can be done this way:

parallel --pipepart -a 001DD.txt --block -1 -k bgzip > 001DD.txt.gz