I try to use my program with a cluster. Now I create small jobs, and submit this to specific nodes with a bash script. These jobs copy a large file to the node, executes 5 other programs on this file and in the end the job copies the file back and deletes it on the node.
This bash script to submit the jobs checks how much hdd is free in a given directory, calculates the maximum number of jobs can be run on this node and sends this number of jobs to this specific node. The disadvantage of this method is that I have to run this bash script multiple times after all jobs have been finished.
Now I want to do this with python with the drmaa api. But now I am wondering if it is able to have node control with this api. Is it possible to submit to a specific node with the drmaa api?
Or do I have to write a workaround with Popen to execute the qsub command and get the number of jobs per node with parsing the output of the qstat command?
I'm not familiar with the drmaa api for Python, but if it provides a way for you to specify limits, you can use a limit to restrict the host that the job will run on. Perhaps use something like the nativeSpecification to pass -l hostname=whatever?
See http://www.gridengine.eu/programming-apis
Setting job submission parameters, which are not defined by the DRMAA standard is easy: They can be set with using the DRMAA standardized native specification, which is in Go the SetNativeSpecification() job template method.