Parallelizing python script with a python wrapper

I have a python script that I have parallelized using GNU Parallel called from a bash wrapper script I use this to process fastq formatted files see example.fastq below. While this works, it is inelegant to require the use of two interpreters and sets of dependencies. I would like to rewrite the bash wrapper script using python while achieving the same parallelization.

example.fastq This is an example of an input file that needs to be processed. This input file is often very long (~500,000,000) lines.

@SRR6750041.1 1/1
@SRR6750041.2 2/1
@SRR6750041.3 3/1
@SRR6750041.4 4/1

Below are minimal reproducible examples of the scripts I am starting out with.

#!/usr/bin/env python
import argparse

# Read in arguments
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--inputFastq', required=True, help='forward .fastq')
parser.add_argument('-o', '--outputFastq', required=True, help='output .fastq')
args = parser.parse_args()

# Iterate through input file and append to output file
with open(args.inputFastq, "r") as infile:
    with open(args.outputFastq, "a") as outfile:
    for line in infile:
        outfile.write("modified" + line)



# split the input fastq for parallel processing. One split fastq file will be created for     each core available.
split --number="l/$NUMCORES" $FASTQ_F split_fastq_F_

# Feed split fastq files to GNU Parallel to invoke parallel executions of ``
ls split_fastq_F* | awk -F "split_fastq_F" '{print $2}' | parallel "python -i split_fastq_F{} -o output.fastq"

#remove intermediate split fastq files
rm split_fastq_*

To execute these scripts I use the command bash You can see that a results file output.fastq is created and contains a modified fastq file.

Below is my attempt to invoke parallel processing using a python wrapper

#!/usr/bin/env python

import heavy_lifting
from joblib import Parallel, delayed
import multiprocessing

numcores = 4
fastq_F = "fastq_F.fastq"

#Create some logic to split the input fastq file into chunks for parallel processing.  

# Get input fastq file dimensions
with open(fastq_F, "r") as infile:
    length_fastq = len(infile.readlines())
    lines = infile.readlines()
    split_size = length_fastq / numcores

# Iterate through input fastq file writing lines to outfile in bins.
counter = 0
split_counter = 0
split_fastq_list = []
with open(fastq_F, "r") as infile:
    for line in infile:
        if counter == 0:
            filename = str("./split_fastq_F_" + str(split_counter))
            outfile = open(filename, "a")
            counter += 1
        elif counter <= split_size:
            counter += 1
            counter = 0
            split_counter += 1

Parallel(n_jobs=numcores)(delayed(heavy_lifting)(i, "output.fastq") for i in split_fastq_list)

EDITED to improve reproducibility of

I seem to be be most confused about how to properly feed the input arguments into the invocation of "Parallel" in the python script. Any help is much appreciated!


  • Parallel expects function's name, not file/module name

    So in heavy_lifting you have to put code in function (with arguments instead of args)

    def my_function(inputFastq, outputFastq):
        with open(inputFastq, "r") as infile:
            with open(outputFastq, "a") as outfile:
                for line in infile:
                    outfile.write("modified" + line)

    And then you can use

    Parallel(n_jobs=numcores)(delayed(heavy_lifting.my_function)(i, "output.fastq") for i in split_fastq_list)