I notice that my backup rsync script spends quite some time copying stuff with random name from .snakemake/metadata folders.
What are those files used for?
Can I safely erase them after a snakemake run has completed, or are they necessary for snakemake to correctly perform the next run?
More generally, is there some documentation about the files that snakemake creates in the .snakemake folder?
From this comment by Johannes Koster, creator of Snakemake:
[The .snakemake/ directory] is used to track (a) the value of the version keyword for each file, (b) the rule implementation for each file, in order to notify the user if something has changed when snakemake is invoked with --summary.
From a related comment on the Google Group:
In general, it is safe to delete the entire .snakemake directory if there is no running Snakemake instance and you are sure that all existing output files are complete. It only contains data provenance information (e.g., to track code input file or parameter changes [to determine if the workflow should be re-run]). You might want to keep .snakemake/conda, since it contains the conda environments used in your workflow.
Edit: To automatically remove the .snakemake/ directory upon successful execution of the pipeline, the onssuccess hook can be used:
import shutil
onsuccess:
shutil.rmtree(".snakemake")