Mix shell command in environment conda with python scripts in snakemake tool

Question

I would like to mix shell command in environment conda with any Python scripts, so the use of the 'run' section isn't possible ...

I tried :

shell:"""
   gunzip -c {input.ech} | NanoFilt -l {params.min_length} --maxlength {params.max_length} -s {input.summary} -q {params.q} --readtype {params.rd} | gzip > Working_space_ont/01_nanofilt/{wildcards.sample}_filt_{params.q}.fastq.gz   
   python3 reinit.py
   while [[ -f Working_space_ont/01_nanofilt/{wildcards.sample}_filt_{params.q}.fastq.gz ]] && [[ grep -c '>' Working_space_ont/01_nanofilt/{wildcards.sample}_filt_{params.q}.fastq.gz < {params.cov} ]]; do
       python3 quality_minor.py
       gunzip -c {input.ech} | NanoFilt -l {params.min_length} --maxlength {params.max_length} -s {input.summary} -q {params.q} --readtype {params.rd} | gzip > Working_space_ont/01_nanofilt/{wildcards.sample}_filt_{params.q}.fastq.gz
   done
   mv Working_space_ont/01_nanofilt/{wildcards.sample}_filt_{params.q}.fastq.gz {output} 
"""

I have non-zero exit status 1 error

reinit.py :

import yaml

with open("config_wf.yaml") as f:
    old_yaml = yaml.load(f)

old_yaml["params"]["filtration"]["quality"] = old_yaml["params"]["filtration"]["quality_fix"]  

with open("config_wf.yaml", 'w') as f:
    yaml.dump(old_yaml, f, default_flow_style=False)

quality_minor.py :

import yaml

with open("config_wf.yaml") as f:
    old_yaml = yaml.load(f)

old_yaml["params"]["filtration"]["quality"] -= 1

with open("config_wf.yaml", 'w') as f:
    yaml.safe(old_yaml, f, default_flow_style=False)

Do you have any ideas for me?
Best Regards,

Eva

Answer 1

It looks like your python scripts are altering the config yaml for the workflow and not passing that directly to the nanofilt command. Are you getting an infinite loop then? The config yaml is read at the start the snakemake process so altering it in a rule should have no effect.

You may be able to achieve this through snakemake using attempts in an input function to the params where each failure decrements the parameter and a job fails when the filtering is bad.

A cleaner solution would be to write a python script that would use nanofilt as a library and wrap its execution. The main method is straight forward. Build up your args and adjust as needed then all the evaluation code and loops will be in python.