I would like to concatenate multiple csv files results from the program ABRICATE (github.com/tseemann/abricate) into a single csv file.
The following snakefile runs through a folder of fasta files producing an individual csv for each fasta. I would like to concatenate those results into a single csv file, but not sure how to frame "rule all".
Any suggestions?
Below is my working snakefile. Thanks
########################### snakefile making dictionary
import os
import yaml
file_list = []
### location assumes that data is in results_fasta/ folder
for entry in os.scandir("results_fasta/"):
if entry.is_file():
file_list.append(entry.name)
#### this tells split the file for use in dictionary creation
config_dict = {"samples":{i.split(".")[0]:"results_fasta/"+i for i in file_list}}
with open("config_abricate_resfinder.yaml","w") as handle:
yaml.dump(config_dict,handle)
###### dictionary created using all files located in results_fasta folder
configfile: "config_abricate_resfinder.yaml"
##### not needed but used to say what is currently running
print("Starting abricate workflow")
##### rule all is a general rule that says this is the results we are lookin for in the end.
rule all:
input:
expand("abricate_resfinder/{sample}_abricate_resfinder.csv", sample = config["samples"])
##### Abricate resfinder
rule resfinder:
input:
lambda wildcards: config["samples"][wildcards.sample]
params:
db_resfinder = "leaveblank", ### for resfinder DB is default so leave blank
type = "csv"
output:
res = "abricate_resfinder/{sample}_abricate_resfinder.csv",
######## log file is empty. need to print stdout to log
log:
"logs/{sample}_resfinder.log"
shell:
"abricate {input} --{params.type} > {output.res}"
If the files have no headers and the fields match, the concatenation may be done by cat
command:
rule all:
input:
expand("abricate_resfinder/{sample}_abricate_resfinder.csv", sample = config["samples"])
output:
out.csv
shell:
"cat {input} > {output}"
If there are headers, you need to think first what result you actually need. The simplest would be to remove all headers at all. You may try this:
"cat {input} | sed '/^#/d' > {output}"