Edit, before trying anything, make sure you install Snakemake with:
conda install -c bioconda -c conda-forge snakemake
As advertised here: snakemake.readthedocs.io. Don't install it as advertised here: anaconda.org/bioconda/snakemake, you will end up with a very old version (the -c conda-forge is important!)
Original post =>
I have been wrestling with Snakemake today. My problem is that my STAR rule gives me an error:
/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/activate-binutils_linux-64.sh: line 67: HOST: unbound variable
Error in job star_map while creating output file /rst1/2017-0205_illuminaseq/scratch/swo-406/preprocessing/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14.Aligned.out.bam.
RuleException:
CalledProcessError in line 50 of /home/nlv24077/experiments/experiments/swo-406/scripts/Snakefile.snakefile:
Command '
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
STAR --runThreadN 8 --genomeDir /rst1/2017-0205_illuminaseq/scratch/swo-390/STAR_references/human --readFilesIn /rst1/2017-0205_illuminaseq/scratch/swo-406/fastq/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14_R1_001.fastq.gz /rst1/2017-0205_illuminaseq/scratch/swo-406/fastq/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14_R2_001.fastq.gz --outSAMtype BAM Unsorted --readFilesCommand zcat --outFileNamePrefix /rst1/2017-0205_illuminaseq/scratch/swo-406/preprocessing/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14.
' returned non-zero exit status 1.
File "/home/nlv24077/experiments/experiments/swo-406/scripts/Snakefile.snakefile", line 50, in __rule_star_map
File "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Exiting because a job execution failed. Look above for error message
However, when I just copy that script/command into a terminal, it works.
This is my snakefile:
import os
from glob import glob
#from snakemake.utils import validate
configfile: 'config.yaml'
#validate(config, "config.schema.yaml")
# Set the working directory
workdir: config['workdir']
experiment_name = 'swo-406'
scratch_data_base_dir="/rst1/2017-0205_illuminaseq/scratch"
scratch_data_dir = os.path.join(scratch_data_base_dir, experiment_name)
seqrun = '180413_NB501997_0054_AHTFJ3BGX3'
fastq_dir = os.path.join(scratch_data_dir, 'fastq', seqrun)
preprocessing_dir = os.path.join(scratch_data_dir, 'preprocessing', seqrun)
quantification_dir = os.path.join(scratch_data_dir, 'quantification', seqrun)
if not os.path.isdir(preprocessing_dir):
os.makedirs(preprocessing_dir)
#ref_base_dir = config[ref_base_dir]
ref_genome = os.path.join(config['ref_base_dir'], config['ref_genome'])
star_ref_dir = config['star_ref_dir']
## Rsem settings
rsem_ref_dir = os.path.join(scratch_data_base_dir, 'swo-387', 'RSEM_references')
rsem_ref_base = os.path.join(rsem_ref_dir, 'Homo_sapiens.GRCh38')
log = os.path.join(preprocessing_dir, 'log.txt')
SAMPLES = set([os.path.basename(fastq_file.replace('_R1_001.fastq.gz', '').replace('_R2_001.fastq.gz', ''))
for fastq_file in glob(os.path.join(fastq_dir, '*_R*_001.fastq.gz'))
if not 'Undetermined' in fastq_file])
#star_output_prefix = os.path.join(preprocessing_dir, '{sample}.')
# Rule all is a pseudo-rule that tells snakemake what final files to generate.
rule all:
input:
expand(os.path.join(quantification_dir, '{sample}'), sample=SAMPLES)
rule star_map:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz'),
output:
os.path.join(preprocessing_dir, '{sample}.Aligned.out.bam')
shell:
"""
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
STAR \
--runThreadN 8 \
--genomeDir {star_ref_dir} \
--readFilesIn {input} \
--outSAMtype BAM Unsorted \
--readFilesCommand zcat \
--outFileNamePrefix {preprocessing_dir}/{wildcards.sample}.
"""
rule samtools_sort:
input:
os.path.join(preprocessing_dir, '{sample}.Aligned.out.bam')
output:
os.path.join(preprocessing_dir, '{sample}.Aligned.out.sorted.bam')
shell:
"""
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
samtools sort -T {wildcards.sample} -O bam {input} > {output}
"""
rule rsem_quantify:
input:
os.path.join(preprocessing_dir, '{sample}.Aligned.out.sorted.bam')
output:
os.path.join(quantification_dir, '{sample}')
shell:
"""
source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
rsem-calculate-expression \
--paired-end \
--bam \
--num-threads 8 \
--strandedness reverse \
{rsem_ref_base} \
{output}
"""
Can anyone spot the error? By the way, I have to comment out
validate(config, "config.schema.yaml")
Because my snakemake.utils does not seem to have a "validate":
(/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake) 16:40 nlv24077@kavia /rst1/2017-0205_illuminaseq/scratch/swo-406 > python3
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from snakemake.utils import validate
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'validate'
>>>
Highest regards,
Freek.
Could you remove all the source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemakecommands from your shell portions of the different rules in the Snakefile and activate the environment either:
run command source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake before you actually run snakemake on that Snakefile (you can even add a version of snakemake that has validate to this environment). So you can run source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake and then run snakemake.
Create a conda environment file matching that environment and add the conda : path/to/created/env/fileparameter in your rules requiring the environment. Then run snakemake with the --use-conda flag
Since you are using the same environment for all your rules, it is better to use option 1 as option 2 is much slower and will make it unecessarily rule specific.
I can reproduce your error with this example Snakefile:
rule test_activate:
output : "test.txt"
shell: "source activate NGS && conda list > {output}"
I get the same unbound variable error but for a different variable as my environment is different. This is an explanation of what might be going on:
Virtualenv activate script won't run in bash script with set -euo
in the sense that once you run it through snakemake vs terminal some variables become unbound which is treated as error.