Search code examples
pythongridcondor

HTCondor output files: obtain created directory

I am using HTcondor to generate some data (txt, png). By running my program, it creates a directory next to the .sub file, named datasets, where the datasets are stored into. Unfortunately, condor does not give me back this created data when finished. In other words, my goal is to get the created data in a "Datasets" subfolder next to the .sub file.

I tried: 1) to not put the data under the datasets subfolder, and I obtained them as thought. Howerver, this is not a smooth solution, since I generate like 100 files which are now mixed up with the .sub file and all the other.

2) Also I tried to set this up in the sub file, leading to this: 

notification = Always
should_transfer_files = YES
RunAsOwner = True
When_To_Transfer_Output = ON_EXIT_OR_EVICT
getenv = True

transfer_input_files = main.py
transfer_output_files = Datasets

universe      = vanilla
log           = log/test-$(Cluster).log
error         = log/test-$(Cluster)-$(Process).err
output        = log/test-$(Cluster)-$(Process).log
executable    = Simulation.bat

queue

This time I get the error, that Datasets was not found. Spelling was checked already.

3) Another option would be, to pack everything in a zip, but since I have to run hundreds of jobs, I do not want to unpack all this files afterwards.

I hope somebody comes up with a good idea on how to solve this.

Solution

  • Just for the record here: HTCondor does not transfer created directories at the end of the run or its contents. The best way to get the content back is to write a wrapper script that will run your executable and then compress the created directory at the root of the working directory. This file will be transferred with all other files. For example, create run.exe:

    ./Simulation.bat
tar zcf Datasets.tar.gz Datasets

    and in your condor submission script put:

    executable    = run.exe

    However, if you do not want to do this and if HTCondor is using a common shared space like an AFS you can simply copy the whole directory out:

    ./Simulation.bat
cp -r Datasets <AFS location>

    The other alternative is to define an initialdir as described at the end of: https://research.cs.wisc.edu/htcondor/manual/quickstart.html

    But one must create the directory structure by hand.

    also, look around pg. 65 of: https://indico.cern.ch/event/611296/contributions/2604376/attachments/1471164/2276521/TannenbaumT_UserTutorial.pdf

    This document is, in general, a very useful one for beginners.