Search code examples
pythoncsvsnakemake

concatenation of ABRICATE csv results in snakemake file


I would like to concatenate multiple csv files results from the program ABRICATE (github.com/tseemann/abricate) into a single csv file.

The following snakefile runs through a folder of fasta files producing an individual csv for each fasta. I would like to concatenate those results into a single csv file, but not sure how to frame "rule all".

Any suggestions?
Below is my working snakefile. Thanks

########################### snakefile making dictionary
import os
import yaml

file_list = []
### location assumes that data is in results_fasta/ folder
for entry in os.scandir("results_fasta/"):
    if entry.is_file():
        file_list.append(entry.name)

#### this tells split the file for use in dictionary creation
config_dict = {"samples":{i.split(".")[0]:"results_fasta/"+i for i in file_list}}

with open("config_abricate_resfinder.yaml","w") as handle:
    yaml.dump(config_dict,handle)

###### dictionary created using all files located in results_fasta folder
configfile: "config_abricate_resfinder.yaml"

##### not needed but used to say what is currently running
print("Starting abricate workflow")

##### rule all is a general rule that says this is the results we are lookin for in the end.
rule all:
    input:
        expand("abricate_resfinder/{sample}_abricate_resfinder.csv", sample = config["samples"])

##### Abricate resfinder
rule resfinder:
    input:
        lambda wildcards: config["samples"][wildcards.sample]
    params:
        db_resfinder = "leaveblank",  ### for resfinder DB is default so leave blank
        type = "csv"
    output:
        res = "abricate_resfinder/{sample}_abricate_resfinder.csv",
######## log file is empty.  need to print stdout to log
    log:
        "logs/{sample}_resfinder.log"
    shell:
        "abricate {input} --{params.type} > {output.res}"

Solution

  • If the files have no headers and the fields match, the concatenation may be done by cat command:

    rule all:
        input:
            expand("abricate_resfinder/{sample}_abricate_resfinder.csv", sample = config["samples"])
        output:
            out.csv
        shell:
            "cat {input} > {output}"
    

    If there are headers, you need to think first what result you actually need. The simplest would be to remove all headers at all. You may try this:

        "cat {input} | sed '/^#/d' > {output}"